[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:35:17 dist-test-slave-kdrb krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:35:17 dist-test-slave-kdrb krb5kdc[399](info): set up 2 sockets
Jun 19 12:35:17 dist-test-slave-kdrb krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 12:35:18.041000 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:35:18 dist-test-slave-kdrb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:35:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:35:18.167374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38947
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:18.278654 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:18.279004 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:18.279091 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:18.283272 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:18.283382 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:18.283420 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:18.283463 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:18.283524 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:18.289053 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:18.290230 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:18.291376 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:18.300402 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:18.300385 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:18.300624 412 server_base.cc:1034] running on GCE node
W20260619 12:35:18.300434 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:18.301548 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:18.302743 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:18.303931 412 hybrid_clock.cc:648] HybridClock initialized: now 1781872518303862 us; error 98 us; skew 500 ppm
Jun 19 12:35:18 dist-test-slave-kdrb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:18.306713 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:35:18.307713 412 webserver.cc:460] Webserver started at http://127.0.98.62:42951/ using document root <none> and password file <none>
I20260619 12:35:18.308303 412 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:18.308404 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:18.308642 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:18.310741 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "86bced7854284736a634d2ec60eb2c23"
format_stamp: "Formatted at 2026-06-19 12:35:18 on dist-test-slave-kdrb"
server_key: "7edda848b059f36db64a0b55b0f81161"
server_key_iv: "8aa56eedbac52623cf42503b21ab21be"
server_key_version: "encryptionkey@0"
I20260619 12:35:18.311350 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "86bced7854284736a634d2ec60eb2c23"
format_stamp: "Formatted at 2026-06-19 12:35:18 on dist-test-slave-kdrb"
server_key: "7edda848b059f36db64a0b55b0f81161"
server_key_iv: "8aa56eedbac52623cf42503b21ab21be"
server_key_version: "encryptionkey@0"
I20260619 12:35:18.315454 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 12:35:18.318075 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:18.319372 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:35:18.319561 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "86bced7854284736a634d2ec60eb2c23"
format_stamp: "Formatted at 2026-06-19 12:35:18 on dist-test-slave-kdrb"
server_key: "7edda848b059f36db64a0b55b0f81161"
server_key_iv: "8aa56eedbac52623cf42503b21ab21be"
server_key_version: "encryptionkey@0"
I20260619 12:35:18.319693 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:18.336134 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:18.339195 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:18.339416 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:18.346688 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38947
I20260619 12:35:18.346714 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38947 every 8 connection(s)
I20260619 12:35:18.347776 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:18.350565 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:18.354176 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 412
I20260619 12:35:18.354303 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:18.355624 392 external_mini_cluster.cc:1428] Setting key 54f782629a73d9479c60217f9ad23b4b
I20260619 12:35:18.356256 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23: Bootstrap starting.
I20260619 12:35:18.359056 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:18.360044 480 log.cc:824] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:18.362068 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23: No bootstrap required, opened a new log
I20260619 12:35:18.366199 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bced7854284736a634d2ec60eb2c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38947 } }
I20260619 12:35:18.366420 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:18.366456 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bced7854284736a634d2ec60eb2c23, State: Initialized, Role: FOLLOWER
I20260619 12:35:18.366963 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bced7854284736a634d2ec60eb2c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38947 } }
I20260619 12:35:18.367089 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:18.367175 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:18.367292 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:18.368307 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bced7854284736a634d2ec60eb2c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38947 } }
I20260619 12:35:18.368649 480 leader_election.cc:302] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86bced7854284736a634d2ec60eb2c23; no voters:
I20260619 12:35:18.368978 480 leader_election.cc:288] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:18.369064 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:18.369252 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [term 1 LEADER]: Becoming Leader. State: Replica: 86bced7854284736a634d2ec60eb2c23, State: Running, Role: LEADER
I20260619 12:35:18.369593 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bced7854284736a634d2ec60eb2c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38947 } }
I20260619 12:35:18.370275 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:18.370782 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86bced7854284736a634d2ec60eb2c23. Latest consensus state: current_term: 1 leader_uuid: "86bced7854284736a634d2ec60eb2c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bced7854284736a634d2ec60eb2c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38947 } } }
I20260619 12:35:18.370914 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:18.371635 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86bced7854284736a634d2ec60eb2c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bced7854284736a634d2ec60eb2c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38947 } } }
I20260619 12:35:18.371733 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:18.372478 489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 12:35:18 dist-test-slave-kdrb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:18.375840 489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:18.381562 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:18.358509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.0.98.62:38947)
0619 12:35:18.358940 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:18.358954 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:18.360254 (+ 1300us) server_negotiation.cc:407] Connection header received
0619 12:35:18.364389 (+ 4135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:18.364409 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:18.364736 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:18.365093 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:18.367586 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:18.368614 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:18.370126 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:18.370821 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:18.370978 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:18.375745 (+ 4767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:18.375751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:18.375763 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:18.375792 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:18.377590 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:18.378159 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:18.378164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:18.378169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:18.378283 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:18.378640 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:18.378643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:18.378645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:18.379068 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:18.379275 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:18.379901 (+ 626us) server_negotiation.cc:299] Negotiation successful
0619 12:35:18.380178 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":146,"threads_started":1}
I20260619 12:35:18.382562 489 catalog_manager.cc:1347] Generated new cluster ID: ab0482fd26f044fea83b4e37b8da5717
I20260619 12:35:18.382690 489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:18.419548 489 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:18.420904 489 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:18.431226 489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86bced7854284736a634d2ec60eb2c23: Generated new TSK 0
I20260619 12:35:18.432054 489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:35:18.473065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38947
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:18.598516 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:18.598862 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:18.598912 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:18.602870 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:18.602931 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:18.602991 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:35:18.608263 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38947
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:18.609582 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:18.610625 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:18.620133 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:18.620142 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:18.620142 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:18.620369 506 server_base.cc:1034] running on GCE node
I20260619 12:35:18.620910 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:18.621589 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:18.622772 506 hybrid_clock.cc:648] HybridClock initialized: now 1781872518622756 us; error 35 us; skew 500 ppm
Jun 19 12:35:18 dist-test-slave-kdrb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:18.625976 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:35:18.626914 506 webserver.cc:460] Webserver started at http://127.0.98.1:45781/ using document root <none> and password file <none>
I20260619 12:35:18.627571 506 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:18.627640 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:18.627851 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:18.629907 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "435e0c9fbacc404eb6143fc9ed616a5c"
format_stamp: "Formatted at 2026-06-19 12:35:18 on dist-test-slave-kdrb"
server_key: "d9cf5b45ae1a85a084e2eaa747151095"
server_key_iv: "a06506b42776ba62f02fbaaecb7204e1"
server_key_version: "encryptionkey@0"
I20260619 12:35:18.630492 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "435e0c9fbacc404eb6143fc9ed616a5c"
format_stamp: "Formatted at 2026-06-19 12:35:18 on dist-test-slave-kdrb"
server_key: "d9cf5b45ae1a85a084e2eaa747151095"
server_key_iv: "a06506b42776ba62f02fbaaecb7204e1"
server_key_version: "encryptionkey@0"
I20260619 12:35:18.634295 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 12:35:18.636818 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:18.638065 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:35:18.638197 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "435e0c9fbacc404eb6143fc9ed616a5c"
format_stamp: "Formatted at 2026-06-19 12:35:18 on dist-test-slave-kdrb"
server_key: "d9cf5b45ae1a85a084e2eaa747151095"
server_key_iv: "a06506b42776ba62f02fbaaecb7204e1"
server_key_version: "encryptionkey@0"
I20260619 12:35:18.638316 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:18.653676 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:18.656631 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:18.656836 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:18.657402 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:18.658210 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:18.658286 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:18.658375 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:18.658430 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:18.668152 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39711
I20260619 12:35:18.668186 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39711 every 8 connection(s)
I20260619 12:35:18.669059 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:35:18.670094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 506
I20260619 12:35:18.670208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:35:18.670440 392 external_mini_cluster.cc:1428] Setting key f3e5716f8430af8aaec8c08d6d3f3abf
Jun 19 12:35:18 dist-test-slave-kdrb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:35:18.684760 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:18.670992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55815 (local address 127.0.98.62:38947)
0619 12:35:18.671142 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:18.671146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:18.672250 (+ 1104us) server_negotiation.cc:407] Connection header received
0619 12:35:18.673638 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:18.673644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:18.673711 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:18.673808 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:18.675389 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:18.676356 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:18.677497 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:18.677945 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:18.678074 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:18.680819 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:18.680822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:18.680824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:18.680848 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:18.682291 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:18.682911 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:18.682914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:18.682915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:18.682999 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:18.683461 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:18.683495 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:18.683499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:18.683740 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:18.683824 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:18.684521 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 12:35:18.684633 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 12:35:18.685758 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:18.671528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38947 (local address 127.0.98.1:55815)
0619 12:35:18.672081 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:18.672156 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:18.673418 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:18.674039 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:18.674050 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:18.674591 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:18.675236 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:18.675251 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:18.676496 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:18.676499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:18.677362 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:18.677370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:18.678127 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:18.678130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:18.678281 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:18.678932 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:18.678949 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:18.680645 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:18.682427 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:18.682432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:18.682446 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:18.682820 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:18.683109 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:18.683112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:18.683116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:18.683301 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:18.683852 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:18.683858 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:18.684252 (+ 394us) client_negotiation.cc:769] Sending connection context
0619 12:35:18.684483 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 12:35:18.684876 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":166,"threads_started":1}
I20260619 12:35:18.687069 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38947
I20260619 12:35:18.687326 635 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:18.687886 635 heartbeater.cc:511] Master 127.0.98.62:38947 requested a full tablet report, sending...
I20260619 12:35:18.689620 444 ts_manager.cc:194] Registered new tserver with Master: 435e0c9fbacc404eb6143fc9ed616a5c (127.0.98.1:39711)
I20260619 12:35:18.691290 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:35:18.702332 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38947
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:18.823890 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:18.824203 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:18.824303 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:18.828692 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:18.828786 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:18.828884 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:35:18.834606 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38947
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:18.835845 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:18.836887 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:18.847172 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:18.847187 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:18.847178 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:18.847234 640 server_base.cc:1034] running on GCE node
I20260619 12:35:18.847864 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:18.848479 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:18.849638 640 hybrid_clock.cc:648] HybridClock initialized: now 1781872518849606 us; error 48 us; skew 500 ppm
Jun 19 12:35:18 dist-test-slave-kdrb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:18.852721 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:35:18.853681 640 webserver.cc:460] Webserver started at http://127.0.98.2:33385/ using document root <none> and password file <none>
I20260619 12:35:18.854288 640 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:18.854378 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:18.854638 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:18.856928 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "ee73a4f92fc44e288db88ac5c392f448"
format_stamp: "Formatted at 2026-06-19 12:35:18 on dist-test-slave-kdrb"
server_key: "b2041121a3af4549b331ef18700b46dc"
server_key_iv: "850804230efb8fb5e0d47d1b2cce08f4"
server_key_version: "encryptionkey@0"
I20260619 12:35:18.857542 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ee73a4f92fc44e288db88ac5c392f448"
format_stamp: "Formatted at 2026-06-19 12:35:18 on dist-test-slave-kdrb"
server_key: "b2041121a3af4549b331ef18700b46dc"
server_key_iv: "850804230efb8fb5e0d47d1b2cce08f4"
server_key_version: "encryptionkey@0"
I20260619 12:35:18.861622 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:35:18.864377 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:18.865749 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:35:18.865881 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "ee73a4f92fc44e288db88ac5c392f448"
format_stamp: "Formatted at 2026-06-19 12:35:18 on dist-test-slave-kdrb"
server_key: "b2041121a3af4549b331ef18700b46dc"
server_key_iv: "850804230efb8fb5e0d47d1b2cce08f4"
server_key_version: "encryptionkey@0"
I20260619 12:35:18.866051 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:18.928369 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:18.932253 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:18.932420 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:18.933102 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:18.933919 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:18.933965 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:18.934010 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:18.934031 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:18.944902 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34539
I20260619 12:35:18.944967 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34539 every 8 connection(s)
I20260619 12:35:18.946005 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:35:18.950139 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 640
I20260619 12:35:18.950268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:35:18.950507 392 external_mini_cluster.cc:1428] Setting key 982e3b0b89856f63991bc5325a216cf6
Jun 19 12:35:18 dist-test-slave-kdrb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:35:18.961470 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:18.947997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53695 (local address 127.0.98.62:38947)
0619 12:35:18.948152 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:18.948155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:18.949023 (+ 868us) server_negotiation.cc:407] Connection header received
0619 12:35:18.950481 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:18.950485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:18.950551 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:18.950660 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:18.952155 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:18.952694 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:18.954272 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:18.954520 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:18.954642 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:18.957595 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:18.957598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:18.957600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:18.957624 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:18.959070 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:18.959669 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:18.959673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:18.959676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:18.959753 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:18.960132 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:18.960135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:18.960137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:18.960413 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:18.960513 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:18.961177 (+ 664us) server_negotiation.cc:299] Negotiation successful
0619 12:35:18.961305 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 12:35:18.962312 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:18.948363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38947 (local address 127.0.98.2:53695)
0619 12:35:18.948854 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:18.948904 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:18.950201 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:18.950868 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:18.950878 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:18.951315 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:18.951968 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:18.951989 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:18.952827 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:18.952833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:18.954136 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:18.954146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:18.954681 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:18.954685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:18.954858 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:18.955592 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:18.955608 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:18.957372 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:18.959192 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:18.959198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:18.959215 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:18.959558 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:18.959847 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:18.959851 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:18.959853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:18.960043 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:18.960676 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:18.960681 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:18.960928 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 12:35:18.961120 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 12:35:18.961461 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":117,"threads_started":1}
I20260619 12:35:18.963551 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38947
I20260619 12:35:18.963871 769 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:18.964393 769 heartbeater.cc:511] Master 127.0.98.62:38947 requested a full tablet report, sending...
I20260619 12:35:18.965421 444 ts_manager.cc:194] Registered new tserver with Master: ee73a4f92fc44e288db88ac5c392f448 (127.0.98.2:34539)
I20260619 12:35:18.966027 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260619 12:35:19.264232 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.312s user 0.002s sys 0.002s
I20260619 12:35:19.264755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38947
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:19.381178 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:19.381456 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:19.381503 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:19.385620 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:19.385689 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:19.385785 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:35:19.391180 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38947
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:19.392359 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:19.393424 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:19.403105 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:19.403105 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:19.403127 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:19.403739 774 server_base.cc:1034] running on GCE node
I20260619 12:35:19.404156 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:19.404785 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:19.405983 774 hybrid_clock.cc:648] HybridClock initialized: now 1781872519405973 us; error 33 us; skew 500 ppm
Jun 19 12:35:19 dist-test-slave-kdrb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:19.408885 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:35:19.409814 774 webserver.cc:460] Webserver started at http://127.0.98.3:37141/ using document root <none> and password file <none>
I20260619 12:35:19.410552 774 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:19.410660 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:19.410894 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:19.412994 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "9c6843e819d6449ba3cbab2b40373150"
format_stamp: "Formatted at 2026-06-19 12:35:19 on dist-test-slave-kdrb"
server_key: "504d7c2ca70d7987b2fa7a4014220c36"
server_key_iv: "46abd7fdae152be70fec2745e5b56a63"
server_key_version: "encryptionkey@0"
I20260619 12:35:19.413596 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9c6843e819d6449ba3cbab2b40373150"
format_stamp: "Formatted at 2026-06-19 12:35:19 on dist-test-slave-kdrb"
server_key: "504d7c2ca70d7987b2fa7a4014220c36"
server_key_iv: "46abd7fdae152be70fec2745e5b56a63"
server_key_version: "encryptionkey@0"
I20260619 12:35:19.417493 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:35:19.420034 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:19.421276 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:35:19.421414 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "9c6843e819d6449ba3cbab2b40373150"
format_stamp: "Formatted at 2026-06-19 12:35:19 on dist-test-slave-kdrb"
server_key: "504d7c2ca70d7987b2fa7a4014220c36"
server_key_iv: "46abd7fdae152be70fec2745e5b56a63"
server_key_version: "encryptionkey@0"
I20260619 12:35:19.421532 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:19.440289 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:19.443823 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:19.444032 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:19.444628 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:19.445452 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:19.445530 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:19.445621 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:19.445672 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:19.454666 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41469
I20260619 12:35:19.454730 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41469 every 8 connection(s)
I20260619 12:35:19.455725 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:35:19.460919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 774
I20260619 12:35:19.461059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:35:19.461318 392 external_mini_cluster.cc:1428] Setting key 7a6756068d2753ad98d0506a3e08261c
Jun 19 12:35:19 dist-test-slave-kdrb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:19.469863 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:19.457915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46163 (local address 127.0.98.62:38947)
0619 12:35:19.458053 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:19.458056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:19.458812 (+ 756us) server_negotiation.cc:407] Connection header received
0619 12:35:19.460200 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:19.460203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:19.460263 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:19.460371 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:19.461604 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:19.462245 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:19.463750 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:19.464045 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:19.464129 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:19.466498 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:19.466501 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:19.466503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:19.466527 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:19.467911 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:19.468356 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:19.468358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:19.468360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:19.468429 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:19.468708 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:19.468710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:19.468711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:19.468870 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:19.468955 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:19.469522 (+ 567us) server_negotiation.cc:299] Negotiation successful
0619 12:35:19.469643 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 12:35:19.470582 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:19.458240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38947 (local address 127.0.98.3:46163)
0619 12:35:19.458667 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:19.458708 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:19.459994 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:19.460541 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:19.460548 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:19.460894 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:19.461466 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:19.461486 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:19.462418 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:19.462421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:19.463623 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:19.463632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:19.464177 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:19.464180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:19.464319 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:19.464867 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:19.464893 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:19.466338 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:19.468013 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:19.468018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:19.468030 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:19.468274 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:19.468505 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:19.468508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:19.468511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:19.468634 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:19.468980 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:19.468993 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:19.469271 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 12:35:19.469433 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 12:35:19.469684 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":107,"threads_started":1}
I20260619 12:35:19.471827 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38947
I20260619 12:35:19.472051 903 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:19.472623 903 heartbeater.cc:511] Master 127.0.98.62:38947 requested a full tablet report, sending...
I20260619 12:35:19.473685 444 ts_manager.cc:194] Registered new tserver with Master: 9c6843e819d6449ba3cbab2b40373150 (127.0.98.3:41469)
I20260619 12:35:19.474255 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46163
I20260619 12:35:19.475814 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 12:35:19 dist-test-slave-kdrb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 12:35:19 dist-test-slave-kdrb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:35:19.502053 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:19.492072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.0.98.1:39711)
0619 12:35:19.492368 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:19.492379 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:19.492473 (+ 94us) server_negotiation.cc:407] Connection header received
0619 12:35:19.492678 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:19.492684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:19.492877 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:19.493040 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:19.493758 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:19.494501 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:19.495347 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:19.495790 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:19.495911 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:19.498674 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:19.498683 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:19.498696 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:19.498720 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:19.500386 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:19.500869 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:19.500874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:19.500875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:19.500950 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:19.501212 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:19.501214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:19.501215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:19.501443 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:19.501598 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:19.501777 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 12:35:19.501896 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
W20260619 12:35:19.502974 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54446
Jun 19 12:35:19 dist-test-slave-kdrb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 12:35:19 dist-test-slave-kdrb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:35:19.528951 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:19.520279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.0.98.1:39711)
0619 12:35:19.520425 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:19.520430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:19.520612 (+ 182us) server_negotiation.cc:407] Connection header received
0619 12:35:19.520784 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:19.520787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:19.520832 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:19.520915 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:19.521612 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:19.522115 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:19.523030 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:19.523246 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:19.523356 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:19.525851 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:19.525854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:19.525857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:19.525891 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:19.527227 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:19.527728 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:19.527732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:19.527734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:19.527815 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:19.528091 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:19.528095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:19.528096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:19.528269 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:19.528425 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:19.528655 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 12:35:19.528780 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 12:35:19.529978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 506
I20260619 12:35:19.535708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 640
I20260619 12:35:19.543340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 774
I20260619 12:35:19.549214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 412
2026-06-19T12:35:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2572 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:35:19 dist-test-slave-kdrb krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:35:19 dist-test-slave-kdrb krb5kdc[917](info): set up 2 sockets
Jun 19 12:35:19 dist-test-slave-kdrb krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 12:35:20.595185 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:35:20 dist-test-slave-kdrb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:35:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:35:20.714944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36075
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:20.825152 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:20.825495 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:20.825588 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:20.829747 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:20.829849 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:20.829885 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:20.829943 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:20.829982 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:20.836094 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:20.837285 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:20.838388 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:20.847023 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:20.847059 930 server_base.cc:1034] running on GCE node
W20260619 12:35:20.847002 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:20.846997 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:20.847934 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:20.849089 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:20.850275 930 hybrid_clock.cc:648] HybridClock initialized: now 1781872520850253 us; error 37 us; skew 500 ppm
Jun 19 12:35:20 dist-test-slave-kdrb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:20.853212 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:35:20.854151 930 webserver.cc:460] Webserver started at http://127.0.98.62:38317/ using document root <none> and password file <none>
I20260619 12:35:20.854705 930 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:20.854766 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:20.855011 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:20.857123 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "79478befaf3d488f982331f1e3801817"
format_stamp: "Formatted at 2026-06-19 12:35:20 on dist-test-slave-kdrb"
server_key: "36a81692edf572b2ecbde910fff8a066"
server_key_iv: "54f415e3ecfb923082c6f88ca5b783f1"
server_key_version: "encryptionkey@0"
I20260619 12:35:20.857685 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "79478befaf3d488f982331f1e3801817"
format_stamp: "Formatted at 2026-06-19 12:35:20 on dist-test-slave-kdrb"
server_key: "36a81692edf572b2ecbde910fff8a066"
server_key_iv: "54f415e3ecfb923082c6f88ca5b783f1"
server_key_version: "encryptionkey@0"
I20260619 12:35:20.861406 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 12:35:20.863803 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:20.865057 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 12:35:20.865199 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "79478befaf3d488f982331f1e3801817"
format_stamp: "Formatted at 2026-06-19 12:35:20 on dist-test-slave-kdrb"
server_key: "36a81692edf572b2ecbde910fff8a066"
server_key_iv: "54f415e3ecfb923082c6f88ca5b783f1"
server_key_version: "encryptionkey@0"
I20260619 12:35:20.865315 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:20.880359 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:20.883853 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:20.884074 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:20.891323 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36075
I20260619 12:35:20.891350 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36075 every 8 connection(s)
I20260619 12:35:20.892442 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:20.895313 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:20.901157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 930
I20260619 12:35:20.901288 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:20.901295 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817: Bootstrap starting.
I20260619 12:35:20.901549 392 external_mini_cluster.cc:1428] Setting key 1c823cb8c7df5898c697c33ad5d28a4c
I20260619 12:35:20.904042 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:20.904804 998 log.cc:824] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:20.906692 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817: No bootstrap required, opened a new log
I20260619 12:35:20.910143 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79478befaf3d488f982331f1e3801817" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36075 } }
I20260619 12:35:20.910401 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:20.910485 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79478befaf3d488f982331f1e3801817, State: Initialized, Role: FOLLOWER
I20260619 12:35:20.910972 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79478befaf3d488f982331f1e3801817" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36075 } }
Jun 19 12:35:20 dist-test-slave-kdrb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:20.911129 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:20.911208 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:20.911309 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:20.912338 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79478befaf3d488f982331f1e3801817" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36075 } }
I20260619 12:35:20.912711 998 leader_election.cc:302] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79478befaf3d488f982331f1e3801817; no voters:
I20260619 12:35:20.913113 998 leader_election.cc:288] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:20.913223 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:20.913434 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [term 1 LEADER]: Becoming Leader. State: Replica: 79478befaf3d488f982331f1e3801817, State: Running, Role: LEADER
I20260619 12:35:20.913743 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79478befaf3d488f982331f1e3801817" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36075 } }
I20260619 12:35:20.914458 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:20.915537 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79478befaf3d488f982331f1e3801817" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79478befaf3d488f982331f1e3801817" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36075 } } }
I20260619 12:35:20.915657 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:20.915970 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79478befaf3d488f982331f1e3801817. Latest consensus state: current_term: 1 leader_uuid: "79478befaf3d488f982331f1e3801817" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79478befaf3d488f982331f1e3801817" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36075 } } }
I20260619 12:35:20.916090 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:20.917953 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:20.902990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58692 (local address 127.0.98.62:36075)
0619 12:35:20.903450 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:20.903519 (+ 69us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:20.903574 (+ 55us) server_negotiation.cc:407] Connection header received
0619 12:35:20.904341 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:20.904360 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:20.904725 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:20.905142 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:20.905984 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:20.907148 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:20.908179 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:20.908715 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:20.908931 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:20.911810 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:20.911815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:20.911829 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:20.911856 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:20.914015 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:20.914564 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:20.914570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:20.914576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:20.914718 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:20.915052 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:20.915056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:20.915058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:20.915680 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:20.915861 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:20.916579 (+ 718us) server_negotiation.cc:299] Negotiation successful
0619 12:35:20.916925 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":148,"threads_started":1}
W20260619 12:35:20.919761 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:35:20.919855 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:35:20.919941 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:20.920845 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:20.925828 1012 catalog_manager.cc:1347] Generated new cluster ID: 3648ada5a12b49c28ec8603183b0bba0
I20260619 12:35:20.925925 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:20.947821 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:20.948820 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:20.955950 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79478befaf3d488f982331f1e3801817: Generated new TSK 0
I20260619 12:35:20.956681 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:35:20.990859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36075
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:21.113694 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:21.113983 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:21.114029 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:21.118533 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:21.118602 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:21.118700 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:35:21.124104 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36075
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:21.125212 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:21.126268 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:21.136247 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:21.136248 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:21.136561 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:21.136615 1024 server_base.cc:1034] running on GCE node
I20260619 12:35:21.137065 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:21.137769 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:21.138958 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781872521138939 us; error 34 us; skew 500 ppm
Jun 19 12:35:21 dist-test-slave-kdrb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:21.141914 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:35:21.142786 1024 webserver.cc:460] Webserver started at http://127.0.98.1:39105/ using document root <none> and password file <none>
I20260619 12:35:21.143348 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:21.143456 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:21.143716 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:21.146371 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "500ec2558762431cb2cc587f273248f0"
format_stamp: "Formatted at 2026-06-19 12:35:21 on dist-test-slave-kdrb"
server_key: "3f0b7c3fff5b3e836382fa6c562fb0ab"
server_key_iv: "e6845361ef0a4bc5b33af09019043288"
server_key_version: "encryptionkey@0"
I20260619 12:35:21.147135 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "500ec2558762431cb2cc587f273248f0"
format_stamp: "Formatted at 2026-06-19 12:35:21 on dist-test-slave-kdrb"
server_key: "3f0b7c3fff5b3e836382fa6c562fb0ab"
server_key_iv: "e6845361ef0a4bc5b33af09019043288"
server_key_version: "encryptionkey@0"
I20260619 12:35:21.150892 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.001s
I20260619 12:35:21.153395 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:21.154601 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:35:21.154719 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "500ec2558762431cb2cc587f273248f0"
format_stamp: "Formatted at 2026-06-19 12:35:21 on dist-test-slave-kdrb"
server_key: "3f0b7c3fff5b3e836382fa6c562fb0ab"
server_key_iv: "e6845361ef0a4bc5b33af09019043288"
server_key_version: "encryptionkey@0"
I20260619 12:35:21.154830 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:21.169206 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:21.172501 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:21.172716 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:21.173302 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:21.174116 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:21.174161 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:21.174227 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:21.174257 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:21.183328 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39805
I20260619 12:35:21.183434 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39805 every 8 connection(s)
I20260619 12:35:21.184333 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:35:21.187788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 1024
I20260619 12:35:21.187878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:35:21.188133 392 external_mini_cluster.cc:1428] Setting key 15215615d57114a949a8d0467c059a81
Jun 19 12:35:21 dist-test-slave-kdrb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:35:21.199445 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.186167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40633 (local address 127.0.98.62:36075)
0619 12:35:21.186314 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:21.186318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:21.187183 (+ 865us) server_negotiation.cc:407] Connection header received
0619 12:35:21.188772 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:21.188776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:21.188835 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:21.188950 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:21.190328 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.190988 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.192233 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.192695 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.192822 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:21.195627 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:21.195633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:21.195638 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:21.195673 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:21.197375 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:21.197965 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:21.197968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:21.197969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:21.198047 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:21.198355 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:21.198358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:21.198359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:21.198529 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:21.198601 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:21.199197 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 12:35:21.199313 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:35:21.200456 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.186559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36075 (local address 127.0.98.1:40633)
0619 12:35:21.187013 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:21.187056 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:21.188533 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:21.189134 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:21.189141 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:21.189656 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:21.190173 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:21.190188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.191126 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.191130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:21.192101 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:21.192111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.192843 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.192846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:21.192992 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:21.193684 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:21.193700 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:21.195392 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:21.197501 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:21.197506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:21.197518 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:21.197846 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:21.198127 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:21.198130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:21.198131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:21.198270 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:21.198645 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:21.198651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:21.198954 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 12:35:21.199119 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 12:35:21.199404 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":118,"threads_started":1}
I20260619 12:35:21.201748 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36075
I20260619 12:35:21.201998 1153 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:21.202512 1153 heartbeater.cc:511] Master 127.0.98.62:36075 requested a full tablet report, sending...
I20260619 12:35:21.204267 962 ts_manager.cc:194] Registered new tserver with Master: 500ec2558762431cb2cc587f273248f0 (127.0.98.1:39805)
I20260619 12:35:21.205411 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:35:21.216523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36075
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:21.328164 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:21.328493 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:21.328585 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:21.332923 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:21.333009 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:21.333097 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:35:21.338457 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36075
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:21.339651 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:21.340679 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:21.350318 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:21.350383 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:21.350294 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:21.350520 1158 server_base.cc:1034] running on GCE node
I20260619 12:35:21.351099 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:21.351796 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:21.352993 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781872521352959 us; error 51 us; skew 500 ppm
Jun 19 12:35:21 dist-test-slave-kdrb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:21.360541 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:35:21.361449 1158 webserver.cc:460] Webserver started at http://127.0.98.2:44501/ using document root <none> and password file <none>
I20260619 12:35:21.361999 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:21.362085 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:21.362258 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:21.364326 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "35b82714480b48619acd61463003d411"
format_stamp: "Formatted at 2026-06-19 12:35:21 on dist-test-slave-kdrb"
server_key: "c4aa2290a98e1b216ba4535f6f6f9f72"
server_key_iv: "f22ec87bb4691dcfe215aab1c2ad2960"
server_key_version: "encryptionkey@0"
I20260619 12:35:21.364841 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "35b82714480b48619acd61463003d411"
format_stamp: "Formatted at 2026-06-19 12:35:21 on dist-test-slave-kdrb"
server_key: "c4aa2290a98e1b216ba4535f6f6f9f72"
server_key_iv: "f22ec87bb4691dcfe215aab1c2ad2960"
server_key_version: "encryptionkey@0"
I20260619 12:35:21.368558 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:35:21.371114 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:21.372670 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:35:21.372816 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "35b82714480b48619acd61463003d411"
format_stamp: "Formatted at 2026-06-19 12:35:21 on dist-test-slave-kdrb"
server_key: "c4aa2290a98e1b216ba4535f6f6f9f72"
server_key_iv: "f22ec87bb4691dcfe215aab1c2ad2960"
server_key_version: "encryptionkey@0"
I20260619 12:35:21.372942 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:21.416890 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:21.419965 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:21.420172 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:21.420775 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:21.421604 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:21.421674 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:21.421746 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:21.421795 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:21.431231 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41721
I20260619 12:35:21.431248 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41721 every 8 connection(s)
I20260619 12:35:21.432337 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:35:21.433254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 1158
I20260619 12:35:21.433360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:35:21.433611 392 external_mini_cluster.cc:1428] Setting key ee8008ba83a4310b418e79754545b558
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 19 12:35:21 dist-test-slave-kdrb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:21.449798 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.434879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41259 (local address 127.0.98.62:36075)
0619 12:35:21.435009 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:21.435012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:21.435678 (+ 666us) server_negotiation.cc:407] Connection header received
0619 12:35:21.437391 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:21.437396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:21.437459 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:21.437555 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:21.438848 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.439414 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.440688 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.440931 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.441053 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:21.446095 (+ 5042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:21.446106 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:21.446109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:21.446139 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:21.447738 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:21.448278 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:21.448281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:21.448283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:21.448364 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:21.448706 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:21.448712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:21.448713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:21.448895 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:21.449008 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:21.449516 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 12:35:21.449634 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:35:21.450654 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.434924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36075 (local address 127.0.98.2:41259)
0619 12:35:21.435506 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:21.435564 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:21.437165 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:21.437717 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:21.437724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:21.438130 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:21.438642 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:21.438657 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.439628 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.439631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:21.440545 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:21.440554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.441093 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.441097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:21.441296 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:21.441956 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:21.441974 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:21.445822 (+ 3848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:21.447862 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:21.447868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:21.447884 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:21.448158 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:21.448463 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:21.448466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:21.448468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:21.448610 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:21.449006 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:21.449013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:21.449265 (+ 252us) client_negotiation.cc:769] Sending connection context
0619 12:35:21.449475 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 12:35:21.449806 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":178,"threads_started":1}
I20260619 12:35:21.451874 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36075
I20260619 12:35:21.452147 1287 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:21.452639 1287 heartbeater.cc:511] Master 127.0.98.62:36075 requested a full tablet report, sending...
I20260619 12:35:21.453714 962 ts_manager.cc:194] Registered new tserver with Master: 35b82714480b48619acd61463003d411 (127.0.98.2:41721)
I20260619 12:35:21.454301 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:35:21.459888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36075
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:21.572556 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:21.572825 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:21.572878 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:21.577052 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:21.577114 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:21.577175 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:35:21.582409 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36075
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:21.583537 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:21.584523 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:21.593897 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:21.593966 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:21.594048 1292 server_base.cc:1034] running on GCE node
W20260619 12:35:21.593897 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:21.594687 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:21.595350 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:21.596553 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781872521596528 us; error 45 us; skew 500 ppm
Jun 19 12:35:21 dist-test-slave-kdrb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:21.599548 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:35:21.600394 1292 webserver.cc:460] Webserver started at http://127.0.98.3:42357/ using document root <none> and password file <none>
I20260619 12:35:21.600912 1292 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:21.600968 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:21.601133 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:21.603103 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "5a91924ed85a41769cdd67ac9cb38b41"
format_stamp: "Formatted at 2026-06-19 12:35:21 on dist-test-slave-kdrb"
server_key: "11f9ee6d5feaf52e08e054f4c17bf7df"
server_key_iv: "692bc409b2b2041c844068fed1a78f33"
server_key_version: "encryptionkey@0"
I20260619 12:35:21.603698 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5a91924ed85a41769cdd67ac9cb38b41"
format_stamp: "Formatted at 2026-06-19 12:35:21 on dist-test-slave-kdrb"
server_key: "11f9ee6d5feaf52e08e054f4c17bf7df"
server_key_iv: "692bc409b2b2041c844068fed1a78f33"
server_key_version: "encryptionkey@0"
I20260619 12:35:21.607259 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 12:35:21.609678 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:21.610930 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:35:21.611030 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "5a91924ed85a41769cdd67ac9cb38b41"
format_stamp: "Formatted at 2026-06-19 12:35:21 on dist-test-slave-kdrb"
server_key: "11f9ee6d5feaf52e08e054f4c17bf7df"
server_key_iv: "692bc409b2b2041c844068fed1a78f33"
server_key_version: "encryptionkey@0"
I20260619 12:35:21.611116 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:21.637844 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:21.641350 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:21.641505 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:21.642023 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:21.642776 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:21.642820 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:21.642858 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:21.642874 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:21.651943 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45127
I20260619 12:35:21.651957 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45127 every 8 connection(s)
I20260619 12:35:21.652951 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:35:21.656136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 1292
I20260619 12:35:21.656263 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:35:21.656595 392 external_mini_cluster.cc:1428] Setting key 3bd3c44775c0df0422ca7edeeb51ddf5
Jun 19 12:35:21 dist-test-slave-kdrb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:21.666424 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.654794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45519 (local address 127.0.98.62:36075)
0619 12:35:21.654944 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:21.654947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:21.655823 (+ 876us) server_negotiation.cc:407] Connection header received
0619 12:35:21.657483 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:21.657487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:21.657543 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:21.657657 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:21.659138 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.659751 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.660835 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.661042 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.661113 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:21.663441 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:21.663444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:21.663446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:21.663465 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:21.664702 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:21.665145 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:21.665148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:21.665149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:21.665218 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:21.665510 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:21.665512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:21.665513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:21.665669 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:21.665741 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:21.666209 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 12:35:21.666305 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:35:21.667364 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.655198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36075 (local address 127.0.98.3:45519)
0619 12:35:21.655681 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:21.655723 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:21.657272 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:21.657846 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:21.657856 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:21.658379 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:21.659015 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:21.659031 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.659854 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.659856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:21.660735 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:21.660744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.661160 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.661162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:21.661295 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:21.661904 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:21.661921 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:21.663302 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:21.664798 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:21.664802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:21.664814 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:21.665063 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:21.665292 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:21.665294 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:21.665295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:21.665426 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:21.665754 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:21.665759 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:21.665997 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 12:35:21.666156 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 12:35:21.666404 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":118,"threads_started":1}
I20260619 12:35:21.668624 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36075
I20260619 12:35:21.668879 1421 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:21.669564 1421 heartbeater.cc:511] Master 127.0.98.62:36075 requested a full tablet report, sending...
I20260619 12:35:21.670665 961 ts_manager.cc:194] Registered new tserver with Master: 5a91924ed85a41769cdd67ac9cb38b41 (127.0.98.3:45127)
I20260619 12:35:21.671204 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45519
I20260619 12:35:21.682431 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:35:21.692706 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.685921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58694 (local address 127.0.98.62:36075)
0619 12:35:21.686072 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:21.686079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:21.686264 (+ 185us) server_negotiation.cc:407] Connection header received
0619 12:35:21.686364 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:21.686366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:21.686408 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:21.686483 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:21.687128 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.687647 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.688418 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.688624 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.688695 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:21.689980 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:21.689982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:21.689984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:21.690001 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:21.691224 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:21.691643 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:21.691646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:21.691647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:21.691719 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:21.691937 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:21.691939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:21.691940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:21.692141 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:21.692237 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:21.692400 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 12:35:21.692498 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 12:35:21.697561 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:35:21.700585 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:21.715350 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.709930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50482 (local address 127.0.98.1:39805)
0619 12:35:21.710296 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:21.710304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:21.710614 (+ 310us) server_negotiation.cc:407] Connection header received
0619 12:35:21.710710 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:21.710714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:21.710901 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:21.711049 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:21.711274 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.712099 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.713665 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.714481 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.714692 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:21.714849 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:21.715122 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 12:35:21.715223 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":96,"threads_started":1}
I20260619 12:35:21.715337 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.709374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39805 (local address 127.0.0.1:50482)
0619 12:35:21.710037 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:21.710050 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:21.710127 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:21.711044 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:21.711047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:21.711065 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:21.711157 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:21.711164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.712220 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.712223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:21.713545 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:21.713553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.714628 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.714631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:21.714809 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:21.714854 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 12:35:21.714997 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 12:35:21.715165 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":134,"thread_start_us":96,"threads_started":1}
I20260619 12:35:21.715867 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.709268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42868 (local address 127.0.98.2:41721)
0619 12:35:21.709730 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:21.709742 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:21.709766 (+ 24us) server_negotiation.cc:407] Connection header received
0619 12:35:21.709855 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:21.709859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:21.710204 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:21.710719 (+ 515us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:21.710877 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.712025 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.713722 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.714481 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.715390 (+ 909us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:21.715583 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:21.715631 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 12:35:21.715712 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":148,"threads_started":1}
I20260619 12:35:21.716125 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.709851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45127 (local address 127.0.0.1:41838)
0619 12:35:21.710445 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:21.710458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:21.710527 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:21.711208 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:21.711211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:21.711232 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:21.711323 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:21.711332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.712715 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.712720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:21.713876 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:21.713883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.715040 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.715043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:21.715126 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:21.715146 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 12:35:21.715213 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 12:35:21.715273 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":22,"thread_start_us":81,"threads_started":1}
I20260619 12:35:21.715338 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.709123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41721 (local address 127.0.0.1:42868)
0619 12:35:21.709489 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:21.709540 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:21.709673 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:21.710375 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:21.710381 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:21.710406 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:21.710566 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:21.710575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.712171 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.712177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:21.713624 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:21.713634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.714637 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.714640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:21.714777 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:21.714856 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 12:35:21.715013 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 12:35:21.715154 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":133,"thread_start_us":89,"threads_started":1}
I20260619 12:35:21.716601 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.710210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.0.98.3:45127)
0619 12:35:21.710610 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:21.710622 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:21.710644 (+ 22us) server_negotiation.cc:407] Connection header received
0619 12:35:21.710708 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:21.710713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:21.710901 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:21.711049 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:21.711455 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.712575 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.714025 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.714635 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.714768 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:21.714944 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:21.715397 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 12:35:21.715870 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":113,"threads_started":1}
I20260619 12:35:21.718752 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 246ba37aa622430ab87bb6e629dcc01a (DEFAULT_TABLE table=test-table [id=6cccad2bf12d4df0b4020a35f7e3ae8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:21.718601 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 246ba37aa622430ab87bb6e629dcc01a (DEFAULT_TABLE table=test-table [id=6cccad2bf12d4df0b4020a35f7e3ae8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:21.718961 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 246ba37aa622430ab87bb6e629dcc01a (DEFAULT_TABLE table=test-table [id=6cccad2bf12d4df0b4020a35f7e3ae8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:21.719743 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246ba37aa622430ab87bb6e629dcc01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:21.719753 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246ba37aa622430ab87bb6e629dcc01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:21.719775 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246ba37aa622430ab87bb6e629dcc01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:21.725862 1438 tablet_bootstrap.cc:492] T 246ba37aa622430ab87bb6e629dcc01a P 5a91924ed85a41769cdd67ac9cb38b41: Bootstrap starting.
I20260619 12:35:21.727327 1439 tablet_bootstrap.cc:492] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0: Bootstrap starting.
I20260619 12:35:21.727391 1440 tablet_bootstrap.cc:492] T 246ba37aa622430ab87bb6e629dcc01a P 35b82714480b48619acd61463003d411: Bootstrap starting.
I20260619 12:35:21.728185 1438 tablet_bootstrap.cc:654] T 246ba37aa622430ab87bb6e629dcc01a P 5a91924ed85a41769cdd67ac9cb38b41: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:21.729296 1438 log.cc:824] T 246ba37aa622430ab87bb6e629dcc01a P 5a91924ed85a41769cdd67ac9cb38b41: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:21.729532 1439 tablet_bootstrap.cc:654] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:21.729986 1440 tablet_bootstrap.cc:654] T 246ba37aa622430ab87bb6e629dcc01a P 35b82714480b48619acd61463003d411: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:21.730393 1439 log.cc:824] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:21.730721 1440 log.cc:824] T 246ba37aa622430ab87bb6e629dcc01a P 35b82714480b48619acd61463003d411: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:21.731391 1438 tablet_bootstrap.cc:492] T 246ba37aa622430ab87bb6e629dcc01a P 5a91924ed85a41769cdd67ac9cb38b41: No bootstrap required, opened a new log
I20260619 12:35:21.731611 1438 ts_tablet_manager.cc:1397] T 246ba37aa622430ab87bb6e629dcc01a P 5a91924ed85a41769cdd67ac9cb38b41: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:35:21.732465 1439 tablet_bootstrap.cc:492] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0: No bootstrap required, opened a new log
I20260619 12:35:21.732666 1439 ts_tablet_manager.cc:1397] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:35:21.733806 1440 tablet_bootstrap.cc:492] T 246ba37aa622430ab87bb6e629dcc01a P 35b82714480b48619acd61463003d411: No bootstrap required, opened a new log
I20260619 12:35:21.733985 1440 ts_tablet_manager.cc:1397] T 246ba37aa622430ab87bb6e629dcc01a P 35b82714480b48619acd61463003d411: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 12:35:21.734879 1438 raft_consensus.cc:348] T 246ba37aa622430ab87bb6e629dcc01a P 5a91924ed85a41769cdd67ac9cb38b41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b82714480b48619acd61463003d411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41721 } } peers { permanent_uuid: "5a91924ed85a41769cdd67ac9cb38b41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45127 } } peers { permanent_uuid: "500ec2558762431cb2cc587f273248f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260619 12:35:21.735133 1438 raft_consensus.cc:374] T 246ba37aa622430ab87bb6e629dcc01a P 5a91924ed85a41769cdd67ac9cb38b41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:21.735185 1438 raft_consensus.cc:729] T 246ba37aa622430ab87bb6e629dcc01a P 5a91924ed85a41769cdd67ac9cb38b41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a91924ed85a41769cdd67ac9cb38b41, State: Initialized, Role: FOLLOWER
I20260619 12:35:21.735651 1438 consensus_queue.cc:260] T 246ba37aa622430ab87bb6e629dcc01a P 5a91924ed85a41769cdd67ac9cb38b41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b82714480b48619acd61463003d411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41721 } } peers { permanent_uuid: "5a91924ed85a41769cdd67ac9cb38b41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45127 } } peers { permanent_uuid: "500ec2558762431cb2cc587f273248f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260619 12:35:21.736295 1421 heartbeater.cc:503] Master 127.0.98.62:36075 was elected leader, sending a full tablet report...
I20260619 12:35:21.736610 1438 ts_tablet_manager.cc:1428] T 246ba37aa622430ab87bb6e629dcc01a P 5a91924ed85a41769cdd67ac9cb38b41: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:35:21.737703 1440 raft_consensus.cc:348] T 246ba37aa622430ab87bb6e629dcc01a P 35b82714480b48619acd61463003d411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b82714480b48619acd61463003d411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41721 } } peers { permanent_uuid: "5a91924ed85a41769cdd67ac9cb38b41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45127 } } peers { permanent_uuid: "500ec2558762431cb2cc587f273248f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260619 12:35:21.737952 1440 raft_consensus.cc:374] T 246ba37aa622430ab87bb6e629dcc01a P 35b82714480b48619acd61463003d411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:21.738005 1440 raft_consensus.cc:729] T 246ba37aa622430ab87bb6e629dcc01a P 35b82714480b48619acd61463003d411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b82714480b48619acd61463003d411, State: Initialized, Role: FOLLOWER
I20260619 12:35:21.738051 1439 raft_consensus.cc:348] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b82714480b48619acd61463003d411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41721 } } peers { permanent_uuid: "5a91924ed85a41769cdd67ac9cb38b41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45127 } } peers { permanent_uuid: "500ec2558762431cb2cc587f273248f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260619 12:35:21.738271 1439 raft_consensus.cc:374] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:21.738324 1439 raft_consensus.cc:729] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500ec2558762431cb2cc587f273248f0, State: Initialized, Role: FOLLOWER
I20260619 12:35:21.738430 1440 consensus_queue.cc:260] T 246ba37aa622430ab87bb6e629dcc01a P 35b82714480b48619acd61463003d411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b82714480b48619acd61463003d411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41721 } } peers { permanent_uuid: "5a91924ed85a41769cdd67ac9cb38b41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45127 } } peers { permanent_uuid: "500ec2558762431cb2cc587f273248f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260619 12:35:21.738692 1439 consensus_queue.cc:260] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b82714480b48619acd61463003d411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41721 } } peers { permanent_uuid: "5a91924ed85a41769cdd67ac9cb38b41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45127 } } peers { permanent_uuid: "500ec2558762431cb2cc587f273248f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260619 12:35:21.739111 1287 heartbeater.cc:503] Master 127.0.98.62:36075 was elected leader, sending a full tablet report...
I20260619 12:35:21.739351 1440 ts_tablet_manager.cc:1428] T 246ba37aa622430ab87bb6e629dcc01a P 35b82714480b48619acd61463003d411: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 12:35:21.739888 1153 heartbeater.cc:503] Master 127.0.98.62:36075 was elected leader, sending a full tablet report...
I20260619 12:35:21.740593 1439 ts_tablet_manager.cc:1428] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
W20260619 12:35:21.903985 1422 tablet.cc:2369] T 246ba37aa622430ab87bb6e629dcc01a P 5a91924ed85a41769cdd67ac9cb38b41: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:35:21.933650 1288 tablet.cc:2369] T 246ba37aa622430ab87bb6e629dcc01a P 35b82714480b48619acd61463003d411: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:35:21.935672 1154 tablet.cc:2369] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:35:21.995802 1446 raft_consensus.cc:482] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:21.996037 1446 raft_consensus.cc:504] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b82714480b48619acd61463003d411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41721 } } peers { permanent_uuid: "5a91924ed85a41769cdd67ac9cb38b41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45127 } } peers { permanent_uuid: "500ec2558762431cb2cc587f273248f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260619 12:35:21.997200 1446 leader_election.cc:288] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35b82714480b48619acd61463003d411 (127.0.98.2:41721), 5a91924ed85a41769cdd67ac9cb38b41 (127.0.98.3:45127)
I20260619 12:35:22.001336 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.997638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60235 (local address 127.0.98.2:41721)
0619 12:35:21.997777 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:21.997782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:21.998097 (+ 315us) server_negotiation.cc:407] Connection header received
0619 12:35:21.998157 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:21.998160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:21.998214 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:21.998288 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:21.998741 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.999233 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:22.000390 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:22.000782 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:22.000900 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:22.000966 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:22.001141 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 12:35:22.001206 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 12:35:22.001366 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.997516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41721 (local address 127.0.98.1:60235)
0619 12:35:21.997854 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:21.997873 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:21.997993 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:21.998429 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:21.998432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:21.998464 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:21.998614 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:21.998623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.999350 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.999357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:22.000259 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:22.000268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:22.000899 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:22.000902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:22.001000 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:22.001022 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:35:22.001128 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 12:35:22.001196 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":1,"spinlock_wait_cycles":1408,"thread_start_us":120,"threads_started":1}
I20260619 12:35:22.001617 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.998099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42139 (local address 127.0.98.3:45127)
0619 12:35:21.998245 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:21.998251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:21.998431 (+ 180us) server_negotiation.cc:407] Connection header received
0619 12:35:21.998600 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:21.998605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:21.998654 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:21.998700 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:21.999052 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.999618 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:22.000615 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:22.001098 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:22.001214 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:22.001270 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:22.001441 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 12:35:22.001504 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 12:35:22.001633 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:21.997947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45127 (local address 127.0.98.1:42139)
0619 12:35:21.998330 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:21.998344 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:21.998450 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:21.998835 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:21.998837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:21.998857 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:21.998944 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:21.998948 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:21.999722 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:21.999728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:22.000491 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:22.000499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:22.001216 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:22.001219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:22.001319 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:22.001343 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 12:35:22.001428 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 12:35:22.001501 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":93,"threads_started":1}
I20260619 12:35:22.002125 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246ba37aa622430ab87bb6e629dcc01a" candidate_uuid: "500ec2558762431cb2cc587f273248f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b82714480b48619acd61463003d411" is_pre_election: true
I20260619 12:35:22.002147 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246ba37aa622430ab87bb6e629dcc01a" candidate_uuid: "500ec2558762431cb2cc587f273248f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a91924ed85a41769cdd67ac9cb38b41" is_pre_election: true
I20260619 12:35:22.002424 1241 raft_consensus.cc:2454] T 246ba37aa622430ab87bb6e629dcc01a P 35b82714480b48619acd61463003d411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 500ec2558762431cb2cc587f273248f0 in term 0.
I20260619 12:35:22.002432 1375 raft_consensus.cc:2454] T 246ba37aa622430ab87bb6e629dcc01a P 5a91924ed85a41769cdd67ac9cb38b41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 500ec2558762431cb2cc587f273248f0 in term 0.
I20260619 12:35:22.002887 1042 leader_election.cc:302] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b82714480b48619acd61463003d411, 500ec2558762431cb2cc587f273248f0; no voters:
I20260619 12:35:22.003165 1446 raft_consensus.cc:2781] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:22.003242 1446 raft_consensus.cc:482] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:22.003271 1446 raft_consensus.cc:3037] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:22.004331 1446 raft_consensus.cc:504] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b82714480b48619acd61463003d411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41721 } } peers { permanent_uuid: "5a91924ed85a41769cdd67ac9cb38b41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45127 } } peers { permanent_uuid: "500ec2558762431cb2cc587f273248f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260619 12:35:22.004673 1446 leader_election.cc:288] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [CANDIDATE]: Term 1 election: Requested vote from peers 35b82714480b48619acd61463003d411 (127.0.98.2:41721), 5a91924ed85a41769cdd67ac9cb38b41 (127.0.98.3:45127)
I20260619 12:35:22.005075 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246ba37aa622430ab87bb6e629dcc01a" candidate_uuid: "500ec2558762431cb2cc587f273248f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b82714480b48619acd61463003d411"
I20260619 12:35:22.005162 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246ba37aa622430ab87bb6e629dcc01a" candidate_uuid: "500ec2558762431cb2cc587f273248f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a91924ed85a41769cdd67ac9cb38b41"
I20260619 12:35:22.005214 1241 raft_consensus.cc:3037] T 246ba37aa622430ab87bb6e629dcc01a P 35b82714480b48619acd61463003d411 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:22.005295 1375 raft_consensus.cc:3037] T 246ba37aa622430ab87bb6e629dcc01a P 5a91924ed85a41769cdd67ac9cb38b41 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:22.006333 1241 raft_consensus.cc:2454] T 246ba37aa622430ab87bb6e629dcc01a P 35b82714480b48619acd61463003d411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 500ec2558762431cb2cc587f273248f0 in term 1.
I20260619 12:35:22.006669 1042 leader_election.cc:302] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b82714480b48619acd61463003d411, 500ec2558762431cb2cc587f273248f0; no voters:
I20260619 12:35:22.006760 1375 raft_consensus.cc:2454] T 246ba37aa622430ab87bb6e629dcc01a P 5a91924ed85a41769cdd67ac9cb38b41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 500ec2558762431cb2cc587f273248f0 in term 1.
I20260619 12:35:22.006862 1446 raft_consensus.cc:2781] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:22.007099 1446 raft_consensus.cc:686] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [term 1 LEADER]: Becoming Leader. State: Replica: 500ec2558762431cb2cc587f273248f0, State: Running, Role: LEADER
I20260619 12:35:22.007427 1446 consensus_queue.cc:237] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b82714480b48619acd61463003d411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41721 } } peers { permanent_uuid: "5a91924ed85a41769cdd67ac9cb38b41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45127 } } peers { permanent_uuid: "500ec2558762431cb2cc587f273248f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260619 12:35:22.010727 961 catalog_manager.cc:5495] T 246ba37aa622430ab87bb6e629dcc01a P 500ec2558762431cb2cc587f273248f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 500ec2558762431cb2cc587f273248f0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "500ec2558762431cb2cc587f273248f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b82714480b48619acd61463003d411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a91924ed85a41769cdd67ac9cb38b41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "500ec2558762431cb2cc587f273248f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } health_report { overall_health: HEALTHY } } }
Jun 19 12:35:22 dist-test-slave-kdrb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:35:22.080094 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:22.071108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50486 (local address 127.0.98.1:39805)
0619 12:35:22.071279 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:22.071284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:22.071440 (+ 156us) server_negotiation.cc:407] Connection header received
0619 12:35:22.071630 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:22.071633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:22.071688 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:22.071782 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:22.072503 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:22.073046 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:22.073907 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:22.074127 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:22.074242 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:22.076933 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:22.076946 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:22.076962 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:22.076989 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:22.078532 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:22.078935 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:22.078941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:22.078942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:22.079019 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:22.079238 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:22.079240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:22.079241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:22.079541 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:22.079700 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:22.079873 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 12:35:22.079972 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 12:35:22 dist-test-slave-kdrb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872522, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 12:35:22 dist-test-slave-kdrb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872522, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:35:22.107445 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:22.098955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50492 (local address 127.0.98.1:39805)
0619 12:35:22.099120 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:22.099125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:22.099286 (+ 161us) server_negotiation.cc:407] Connection header received
0619 12:35:22.099519 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:22.099523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:22.099580 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:22.099671 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:22.100345 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:22.100868 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:22.101705 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:22.101944 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:22.102064 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:22.104412 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:22.104416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:22.104419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:22.104454 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:22.105756 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:22.106182 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:22.106187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:22.106189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:22.106271 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:22.106555 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:22.106558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:22.106559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:22.106730 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:22.106868 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:22.107153 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 12:35:22.107277 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 12:35:22 dist-test-slave-kdrb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 12:35:22 dist-test-slave-kdrb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:35:22.133154 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:22.124458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50498 (local address 127.0.98.1:39805)
0619 12:35:22.124639 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:22.124644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:22.124771 (+ 127us) server_negotiation.cc:407] Connection header received
0619 12:35:22.124957 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:22.124960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:22.125014 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:22.125103 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:22.125809 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:22.126321 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:22.127180 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:22.127415 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:22.127587 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:22.130004 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:22.130008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:22.130011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:22.130045 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:22.131512 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:22.131950 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:22.131955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:22.131956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:22.132039 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:22.132305 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:22.132310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:22.132312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:22.132491 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:22.132658 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:22.132871 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 12:35:22.132989 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260619 12:35:22.134610 1067 tablet_service.cc:2908] Rejecting scan request for tablet 246ba37aa622430ab87bb6e629dcc01a: Uninitialized: clean time has not yet been initialized
I20260619 12:35:22.136723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 1024
I20260619 12:35:22.142786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 1158
I20260619 12:35:22.148469 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 1292
I20260619 12:35:22.154554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 930
2026-06-19T12:35:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2604 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:35:22 dist-test-slave-kdrb krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:35:22 dist-test-slave-kdrb krb5kdc[1463](info): set up 2 sockets
Jun 19 12:35:22 dist-test-slave-kdrb krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260619 12:35:23.199229 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:35:23 dist-test-slave-kdrb krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:35:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:35:23.320358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33529
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:23.429979 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:23.430266 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:23.430334 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:23.434453 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:23.434516 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 12:35:23.434535 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:23.434552 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:23.434609 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:23.434638 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:23.440168 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33529
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:23.441228 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:23.442298 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:23.450845 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:23.450845 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:23.450953 1476 server_base.cc:1034] running on GCE node
W20260619 12:35:23.450846 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:23.451797 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:23.452877 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:23.454085 1476 hybrid_clock.cc:648] HybridClock initialized: now 1781872523454074 us; error 30 us; skew 500 ppm
Jun 19 12:35:23 dist-test-slave-kdrb krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:23.457082 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:35:23.457981 1476 webserver.cc:460] Webserver started at http://127.0.98.62:40865/ using document root <none> and password file <none>
I20260619 12:35:23.458570 1476 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:23.458662 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:23.458895 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:23.461001 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "8f459e71a25b46118371b7fb144ed5bb"
format_stamp: "Formatted at 2026-06-19 12:35:23 on dist-test-slave-kdrb"
server_key: "57325ab04b25a5d6ec098d9aa45a96c5"
server_key_iv: "190fc7841f4e3f8b0b0bbda8770bd906"
server_key_version: "encryptionkey@0"
I20260619 12:35:23.461566 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f459e71a25b46118371b7fb144ed5bb"
format_stamp: "Formatted at 2026-06-19 12:35:23 on dist-test-slave-kdrb"
server_key: "57325ab04b25a5d6ec098d9aa45a96c5"
server_key_iv: "190fc7841f4e3f8b0b0bbda8770bd906"
server_key_version: "encryptionkey@0"
I20260619 12:35:23.465418 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:35:23.467867 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:23.469112 1476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:35:23.469257 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "8f459e71a25b46118371b7fb144ed5bb"
format_stamp: "Formatted at 2026-06-19 12:35:23 on dist-test-slave-kdrb"
server_key: "57325ab04b25a5d6ec098d9aa45a96c5"
server_key_iv: "190fc7841f4e3f8b0b0bbda8770bd906"
server_key_version: "encryptionkey@0"
I20260619 12:35:23.469373 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:23.503079 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:23.506544 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:23.506762 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:23.514618 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33529
I20260619 12:35:23.514638 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33529 every 8 connection(s)
I20260619 12:35:23.515652 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:23.516760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 1476
I20260619 12:35:23.516868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:23.517153 392 external_mini_cluster.cc:1428] Setting key 7d18709a610f8ffcc623a7b08e70bcef
I20260619 12:35:23.518899 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:23.525454 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb: Bootstrap starting.
Jun 19 12:35:23 dist-test-slave-kdrb krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:23.528242 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:23.529632 1554 log.cc:824] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:23.531864 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb: No bootstrap required, opened a new log
I20260619 12:35:23.531875 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:23.518454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.0.98.62:33529)
0619 12:35:23.518849 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:23.518889 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:23.518964 (+ 75us) server_negotiation.cc:407] Connection header received
0619 12:35:23.519684 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:23.519702 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:23.520024 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:23.520298 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:23.521161 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:23.522148 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:23.523255 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:23.523779 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:23.523894 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:23.526626 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:23.526633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:23.526649 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:23.526684 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:23.528780 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:23.529225 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:23.529231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:23.529237 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:23.529369 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:23.529627 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:23.529630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:23.529631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:23.530134 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:23.530268 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:23.530646 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 12:35:23.530926 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":132,"threads_started":1}
I20260619 12:35:23.535106 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f459e71a25b46118371b7fb144ed5bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33529 } }
I20260619 12:35:23.535320 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:23.535360 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f459e71a25b46118371b7fb144ed5bb, State: Initialized, Role: FOLLOWER
I20260619 12:35:23.535820 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f459e71a25b46118371b7fb144ed5bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33529 } }
I20260619 12:35:23.535940 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:23.535987 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:23.536057 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:23.536903 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f459e71a25b46118371b7fb144ed5bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33529 } }
I20260619 12:35:23.537190 1554 leader_election.cc:302] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f459e71a25b46118371b7fb144ed5bb; no voters:
I20260619 12:35:23.537442 1554 leader_election.cc:288] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:23.537585 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:23.537855 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [term 1 LEADER]: Becoming Leader. State: Replica: 8f459e71a25b46118371b7fb144ed5bb, State: Running, Role: LEADER
I20260619 12:35:23.538182 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f459e71a25b46118371b7fb144ed5bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33529 } }
I20260619 12:35:23.538748 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:23.540045 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f459e71a25b46118371b7fb144ed5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f459e71a25b46118371b7fb144ed5bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33529 } } }
I20260619 12:35:23.540043 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f459e71a25b46118371b7fb144ed5bb. Latest consensus state: current_term: 1 leader_uuid: "8f459e71a25b46118371b7fb144ed5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f459e71a25b46118371b7fb144ed5bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33529 } } }
I20260619 12:35:23.540289 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:23.540362 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:23.540700 1571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:23.543892 1571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:23.549383 1571 catalog_manager.cc:1347] Generated new cluster ID: 4a2ff59a90bb48b281bc8ea78fff535b
I20260619 12:35:23.549460 1571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:23.589635 1571 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:23.590756 1571 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:23.604923 1571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb: Generated new TSK 0
I20260619 12:35:23.605901 1571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:35:23.646926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33529
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:23.760991 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:23.761319 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:23.761407 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:23.765600 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:23.765730 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:23.765811 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:35:23.765898 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:35:23.771452 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:23.772773 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:23.773981 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:23.784451 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:23.784540 1580 server_base.cc:1034] running on GCE node
W20260619 12:35:23.784447 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:23.784447 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:23.785235 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:23.785938 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:23.787128 1580 hybrid_clock.cc:648] HybridClock initialized: now 1781872523787093 us; error 57 us; skew 500 ppm
Jun 19 12:35:23 dist-test-slave-kdrb krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:23.790056 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:35:23.790975 1580 webserver.cc:460] Webserver started at http://127.0.98.1:43321/ using document root <none> and password file <none>
I20260619 12:35:23.791620 1580 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:23.791713 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:23.792132 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:23.794474 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "b225da14c9844021bcb88e91cfb28b01"
format_stamp: "Formatted at 2026-06-19 12:35:23 on dist-test-slave-kdrb"
server_key: "825fbaba69a5c80e9d204b5268303425"
server_key_iv: "109ec308b56b43d31b0c90f625a3e0c8"
server_key_version: "encryptionkey@0"
I20260619 12:35:23.795104 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b225da14c9844021bcb88e91cfb28b01"
format_stamp: "Formatted at 2026-06-19 12:35:23 on dist-test-slave-kdrb"
server_key: "825fbaba69a5c80e9d204b5268303425"
server_key_iv: "109ec308b56b43d31b0c90f625a3e0c8"
server_key_version: "encryptionkey@0"
I20260619 12:35:23.799217 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 12:35:23.801962 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:23.803336 1580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:35:23.803529 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "b225da14c9844021bcb88e91cfb28b01"
format_stamp: "Formatted at 2026-06-19 12:35:23 on dist-test-slave-kdrb"
server_key: "825fbaba69a5c80e9d204b5268303425"
server_key_iv: "109ec308b56b43d31b0c90f625a3e0c8"
server_key_version: "encryptionkey@0"
I20260619 12:35:23.803689 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:23.820490 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:23.823899 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:23.824112 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:23.825712 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:23.825788 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:23.825862 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:23.825911 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:35:23 dist-test-slave-kdrb krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:23.837031 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46409
I20260619 12:35:23.838179 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:35:23.839131 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46409 every 8 connection(s)
I20260619 12:35:23.840286 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:23.826713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34543 (local address 127.0.98.62:33529)
0619 12:35:23.826844 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:23.826848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:23.828110 (+ 1262us) server_negotiation.cc:407] Connection header received
0619 12:35:23.829312 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:23.829315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:23.829377 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:23.829492 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:23.830823 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:23.831364 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:23.832602 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:23.832949 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:23.833076 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:23.836166 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:23.836170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:23.836172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:23.836198 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:23.837888 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:23.838484 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:23.838491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:23.838496 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:23.838602 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:23.839070 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:23.839074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:23.839075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:23.839319 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:23.839436 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:23.840010 (+ 574us) server_negotiation.cc:299] Negotiation successful
0619 12:35:23.840143 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 12:35:23.841176 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:23.827326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33529 (local address 127.0.98.1:34543)
0619 12:35:23.827902 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:23.827981 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:23.829098 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:23.829680 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:23.829691 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:23.830061 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:23.830679 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:23.830694 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:23.831544 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:23.831547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:23.832459 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:23.832469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:23.833127 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:23.833131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:23.833298 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:23.834018 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:23.834034 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:23.835969 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:23.838023 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:23.838027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:23.838039 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:23.838380 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:23.838727 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:23.838730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:23.838732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:23.838962 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:23.839454 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:23.839461 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:23.839782 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 12:35:23.839953 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 12:35:23.840321 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":140,"threads_started":1}
I20260619 12:35:23.842409 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33529
I20260619 12:35:23.842579 1711 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:23.843108 1711 heartbeater.cc:511] Master 127.0.98.62:33529 requested a full tablet report, sending...
I20260619 12:35:23.843825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 1580
I20260619 12:35:23.843946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:35:23.844238 392 external_mini_cluster.cc:1428] Setting key a8759090438fe224b70a6178421a1e0f
I20260619 12:35:23.845430 1507 ts_manager.cc:194] Registered new tserver with Master: b225da14c9844021bcb88e91cfb28b01 (127.0.98.1:46409)
I20260619 12:35:23.846726 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34543
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:35:23.855463 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:23.844750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33529 (local address 127.0.98.1:44519)
0619 12:35:23.845030 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:23.845046 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:23.845131 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:23.845675 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:23.845678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:23.845888 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:23.846004 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:23.846011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:23.847204 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:23.847209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:23.848068 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:23.848078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:23.848728 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:23.848731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:23.848848 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:23.851677 (+ 2829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:23.851690 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:23.852049 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:23.853864 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:23.853867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:23.853870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:23.854126 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:23.854408 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:23.854412 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:23.854414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:23.854502 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:23.854879 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:23.854882 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:23.854991 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 12:35:23.855105 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 12:35:23.855262 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":3,"thread_start_us":109,"threads_started":1}
I20260619 12:35:23.855451 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:23.845353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44519 (local address 127.0.98.62:33529)
0619 12:35:23.845476 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:23.845480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:23.845491 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:35:23.845525 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:23.845528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:23.845571 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:23.845654 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:23.846127 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:23.847036 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:23.848203 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:23.848507 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:23.848581 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:23.852213 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:23.852217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:23.852220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:23.852245 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:23.853736 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:23.854232 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:23.854236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:23.854238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:23.854316 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:23.854594 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:23.854599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:23.854601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:23.854786 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:23.854893 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:23.855161 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 12:35:23.855275 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:35:23.872305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33529
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:23.987340 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:23.987658 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:23.987705 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:23.992017 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:23.992094 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:23.992136 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:35:23.992172 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:35:23.997674 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:23.998817 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:23.999902 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:24.010272 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:24.010286 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:24.010375 1720 server_base.cc:1034] running on GCE node
W20260619 12:35:24.010299 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:24.011029 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:24.011821 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:24.013016 1720 hybrid_clock.cc:648] HybridClock initialized: now 1781872524012988 us; error 38 us; skew 500 ppm
Jun 19 12:35:24 dist-test-slave-kdrb krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:24.016134 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:35:24.017148 1720 webserver.cc:460] Webserver started at http://127.0.98.2:36753/ using document root <none> and password file <none>
I20260619 12:35:24.017755 1720 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:24.017819 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:24.018052 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:24.020303 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "eb99b9027dd740c687a8e9ffdff6ddfa"
format_stamp: "Formatted at 2026-06-19 12:35:24 on dist-test-slave-kdrb"
server_key: "9d4c8a9983240cad540c7e15a6f84c1b"
server_key_iv: "8379c5f6c789272202f7adfc560498a0"
server_key_version: "encryptionkey@0"
I20260619 12:35:24.020876 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eb99b9027dd740c687a8e9ffdff6ddfa"
format_stamp: "Formatted at 2026-06-19 12:35:24 on dist-test-slave-kdrb"
server_key: "9d4c8a9983240cad540c7e15a6f84c1b"
server_key_iv: "8379c5f6c789272202f7adfc560498a0"
server_key_version: "encryptionkey@0"
I20260619 12:35:24.024855 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:35:24.027690 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:24.029127 1720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 12:35:24.029244 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "eb99b9027dd740c687a8e9ffdff6ddfa"
format_stamp: "Formatted at 2026-06-19 12:35:24 on dist-test-slave-kdrb"
server_key: "9d4c8a9983240cad540c7e15a6f84c1b"
server_key_iv: "8379c5f6c789272202f7adfc560498a0"
server_key_version: "encryptionkey@0"
I20260619 12:35:24.029402 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:24.054025 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:24.057155 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:24.057363 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:24.058804 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:24.058893 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:24.058970 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:24.059026 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:35:24 dist-test-slave-kdrb krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:24.070837 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36475
I20260619 12:35:24.072158 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:35:24.073428 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36475 every 8 connection(s)
I20260619 12:35:24.074656 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.059822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47623 (local address 127.0.98.62:33529)
0619 12:35:24.059943 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.059947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.061035 (+ 1088us) server_negotiation.cc:407] Connection header received
0619 12:35:24.062168 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.062171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.062232 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.062348 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:24.064111 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.064656 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.066182 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.066418 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.066555 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.069942 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:24.069946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:24.069949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:24.069981 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:24.071892 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:24.072643 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:24.072650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:24.072652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:24.072751 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:24.073255 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:24.073260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:24.073262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:24.073515 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:24.073613 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.074330 (+ 717us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.074471 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 12:35:24.075423 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.060391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33529 (local address 127.0.98.2:47623)
0619 12:35:24.060881 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.060923 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.061875 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.062596 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.062609 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.063106 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:24.063913 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.063936 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.064789 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.064792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.066042 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.066052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.066620 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.066625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.066793 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.067545 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:24.067566 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:24.069687 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:24.072051 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:24.072079 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:24.072099 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:24.072482 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:24.072869 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:24.072874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:24.072876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:24.073088 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:24.073645 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:24.073651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:24.073960 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.074184 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.074595 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":134,"threads_started":1}
I20260619 12:35:24.076931 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33529
I20260619 12:35:24.077100 1851 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:24.077750 1851 heartbeater.cc:511] Master 127.0.98.62:33529 requested a full tablet report, sending...
I20260619 12:35:24.078819 1508 ts_manager.cc:194] Registered new tserver with Master: eb99b9027dd740c687a8e9ffdff6ddfa (127.0.98.2:36475)
I20260619 12:35:24.079664 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47623
I20260619 12:35:24.080318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 1720
I20260619 12:35:24.080415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:35:24.080674 392 external_mini_cluster.cc:1428] Setting key b766a0b3a90e26877e26543f8cd26631
I20260619 12:35:24.087152 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.079160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33529 (local address 127.0.98.2:39657)
0619 12:35:24.079455 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.079492 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.079588 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.079814 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.079818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.080070 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:24.080199 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.080206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.080977 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.080982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.081635 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.081644 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.082243 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.082247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.082369 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.082893 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:24.082914 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:24.083406 (+ 492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:24.085554 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:24.085557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:24.085558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:24.085766 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:24.086071 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:24.086073 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:24.086075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:24.086145 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:24.086696 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:24.086698 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:24.086781 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.086866 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.087003 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":93,"threads_started":1}
I20260619 12:35:24.087329 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.079288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39657 (local address 127.0.98.62:33529)
0619 12:35:24.079416 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.079420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.079545 (+ 125us) server_negotiation.cc:407] Connection header received
0619 12:35:24.079670 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.079672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.079707 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.079797 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:24.080344 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.080851 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.081800 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.082113 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.082219 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.083605 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:24.083613 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:24.083618 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:24.083653 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:24.085420 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:24.085866 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:24.085872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:24.085877 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:24.085980 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:24.086238 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:24.086242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:24.086246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:24.086594 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:24.086705 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.086964 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.087105 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:35:24.107899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33529
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:24.223007 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:24.223345 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:24.223425 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:24.227639 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:24.227732 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:24.227813 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:35:24.227891 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:35:24.233512 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:24.234668 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:24.235836 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:24.246182 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:24.246251 1860 server_base.cc:1034] running on GCE node
W20260619 12:35:24.246160 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:24.246188 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:24.246847 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:24.247526 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:24.248708 1860 hybrid_clock.cc:648] HybridClock initialized: now 1781872524248677 us; error 45 us; skew 500 ppm
Jun 19 12:35:24 dist-test-slave-kdrb krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:24.251575 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:35:24.252513 1860 webserver.cc:460] Webserver started at http://127.0.98.3:32837/ using document root <none> and password file <none>
I20260619 12:35:24.253075 1860 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:24.253136 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:24.253307 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:24.255419 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "f5ae0e0deb484b0daed055910f7c2155"
format_stamp: "Formatted at 2026-06-19 12:35:24 on dist-test-slave-kdrb"
server_key: "eaea642cfc2f1a0c3d66c84666706229"
server_key_iv: "e698f54fd29b59419a5a1af8649deef3"
server_key_version: "encryptionkey@0"
I20260619 12:35:24.256067 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f5ae0e0deb484b0daed055910f7c2155"
format_stamp: "Formatted at 2026-06-19 12:35:24 on dist-test-slave-kdrb"
server_key: "eaea642cfc2f1a0c3d66c84666706229"
server_key_iv: "e698f54fd29b59419a5a1af8649deef3"
server_key_version: "encryptionkey@0"
I20260619 12:35:24.259931 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:35:24.262429 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:24.263789 1860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:35:24.263931 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "f5ae0e0deb484b0daed055910f7c2155"
format_stamp: "Formatted at 2026-06-19 12:35:24 on dist-test-slave-kdrb"
server_key: "eaea642cfc2f1a0c3d66c84666706229"
server_key_iv: "e698f54fd29b59419a5a1af8649deef3"
server_key_version: "encryptionkey@0"
I20260619 12:35:24.264047 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:24.279868 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:24.282971 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:24.283183 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:24.284825 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:24.284878 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:24.284946 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:24.284977 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:35:24 dist-test-slave-kdrb krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:24.295718 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40477
I20260619 12:35:24.295785 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40477 every 8 connection(s)
I20260619 12:35:24.296662 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:35:24.299785 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.285834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45615 (local address 127.0.98.62:33529)
0619 12:35:24.285995 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.286000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.286858 (+ 858us) server_negotiation.cc:407] Connection header received
0619 12:35:24.288140 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.288147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.288213 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.288325 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:24.289597 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.290511 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.292289 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.292678 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.292810 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.296033 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:24.296036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:24.296038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:24.296065 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:24.297776 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:24.298367 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:24.298369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:24.298371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:24.298449 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:24.298758 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:24.298760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:24.298761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:24.298936 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:24.299010 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.299560 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.299664 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:35:24.300627 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.286217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33529 (local address 127.0.98.3:45615)
0619 12:35:24.286679 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.286731 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.287853 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.288555 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.288569 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.288965 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:24.289457 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.289471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.290687 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.290691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.292145 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.292159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.292853 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.292857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.293057 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.293738 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:24.293764 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:24.295795 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:24.297917 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:24.297923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:24.297939 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:24.298265 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:24.298542 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:24.298544 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:24.298545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:24.298673 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:24.299021 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:24.299025 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:24.299264 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.299416 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.299755 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":141,"threads_started":1}
I20260619 12:35:24.301872 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33529
I20260619 12:35:24.302129 1991 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:24.302634 1991 heartbeater.cc:511] Master 127.0.98.62:33529 requested a full tablet report, sending...
I20260619 12:35:24.303728 1507 ts_manager.cc:194] Registered new tserver with Master: f5ae0e0deb484b0daed055910f7c2155 (127.0.98.3:40477)
I20260619 12:35:24.304428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 1860
I20260619 12:35:24.304461 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45615
I20260619 12:35:24.304563 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:35:24.304873 392 external_mini_cluster.cc:1428] Setting key c0c04e06d6053026174ce26c4c5a4803
I20260619 12:35:24.307408 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:35:24.312285 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.304383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33529 (local address 127.0.98.3:51731)
0619 12:35:24.304962 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.304977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.305079 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.305347 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.305350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.305577 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:24.305689 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.305695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.306475 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.306478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.307166 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.307174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.307705 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.307709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.307800 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.308377 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:24.308389 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:24.308727 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:24.310693 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:24.310695 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:24.310697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:24.310910 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:24.311207 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:24.311210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:24.311211 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:24.311282 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:24.311748 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:24.311750 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:24.311827 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.311986 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.312132 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":336,"threads_started":1}
I20260619 12:35:24.312402 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.304452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51731 (local address 127.0.98.62:33529)
0619 12:35:24.304591 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.304595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.305040 (+ 445us) server_negotiation.cc:407] Connection header received
0619 12:35:24.305162 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.305164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.305230 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.305317 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:24.305795 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.306340 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.307286 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.307584 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.307704 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.308859 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:24.308864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:24.308867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:24.308889 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:24.310576 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:24.311014 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:24.311021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:24.311025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:24.311117 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:24.311368 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:24.311374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:24.311378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:24.311661 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:24.311769 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.312070 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.312210 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 12:35:24 dist-test-slave-kdrb krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872524, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 12:35:24 dist-test-slave-kdrb krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872524, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:24.334964 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.326106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.0.98.62:33529)
0619 12:35:24.326259 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.326263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.326276 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:35:24.326316 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.326319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.326356 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.326454 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:24.327167 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.327768 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.328629 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.328851 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.328974 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.331614 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:24.331617 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:24.331620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:24.331651 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:24.333019 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:24.333441 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:24.333443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:24.333444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:24.333521 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:24.333883 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:24.333885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:24.333887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:24.334179 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:24.334314 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.334604 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.334779 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 12:35:24.338519 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:35:24.342506 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:24.359256 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.352216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40477 (local address 127.0.0.1:50838)
0619 12:35:24.352906 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.352920 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.353025 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.354275 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.354278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.354296 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:24.354397 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.354403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.355796 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.355800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.357406 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.357421 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.358380 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.358384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.358499 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.358539 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.358951 (+ 412us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.359086 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":112,"thread_start_us":109,"threads_started":1}
I20260619 12:35:24.359763 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.353329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.0.98.3:40477)
0619 12:35:24.353836 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.353853 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.353877 (+ 24us) server_negotiation.cc:407] Connection header received
0619 12:35:24.353959 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.353964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.354130 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.354292 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:24.354525 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.355690 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.357552 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.358252 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.358380 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.358581 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.359507 (+ 926us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.359620 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":111,"threads_started":1}
I20260619 12:35:24.360106 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.352008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36475 (local address 127.0.0.1:39772)
0619 12:35:24.352799 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.352859 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.353001 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.354008 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.354013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.354034 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:24.354148 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.354153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.355891 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.355895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.357763 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.357768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.359068 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.359075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.359205 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.359246 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.359299 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.359708 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":91,"thread_start_us":92,"threads_started":1}
I20260619 12:35:24.360385 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.352008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46409 (local address 127.0.0.1:46560)
0619 12:35:24.353394 (+ 1386us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.353406 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.353482 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.354843 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.354846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.354863 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:24.354957 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.354963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.356343 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.356347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.357634 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.357645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.360010 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.360014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.360122 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.360145 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.360192 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.360256 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1331,"mutex_wait_us":683,"thread_start_us":98,"threads_started":1}
I20260619 12:35:24.361004 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.352163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.0.98.1:46409)
0619 12:35:24.354481 (+ 2318us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.354489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.354512 (+ 23us) server_negotiation.cc:407] Connection header received
0619 12:35:24.354581 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.354585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.354720 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.354871 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:24.355088 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.356215 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.358644 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.359859 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.360651 (+ 792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.360787 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.360829 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.360899 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2185,"thread_start_us":107,"threads_started":1}
I20260619 12:35:24.362216 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.352259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39772 (local address 127.0.98.2:36475)
0619 12:35:24.353321 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.353334 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.353359 (+ 25us) server_negotiation.cc:407] Connection header received
0619 12:35:24.353454 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.353461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.353699 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.353856 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:24.354290 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.355624 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.357881 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.358914 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.360585 (+ 1671us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.361995 (+ 1410us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.362038 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.362112 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":118,"threads_started":1}
I20260619 12:35:24.363030 1918 tablet_service.cc:1467] Processing CreateTablet for tablet 6da6b3bf75684d37b4830f6285adb078 (DEFAULT_TABLE table=test-table [id=d58bacf450c543f49413aaf388c0df98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:24.363763 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 6da6b3bf75684d37b4830f6285adb078 (DEFAULT_TABLE table=test-table [id=d58bacf450c543f49413aaf388c0df98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:24.364246 1918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da6b3bf75684d37b4830f6285adb078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:24.364811 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da6b3bf75684d37b4830f6285adb078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:24.364665 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 6da6b3bf75684d37b4830f6285adb078 (DEFAULT_TABLE table=test-table [id=d58bacf450c543f49413aaf388c0df98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:24.365542 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da6b3bf75684d37b4830f6285adb078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:24.372341 2014 tablet_bootstrap.cc:492] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01: Bootstrap starting.
I20260619 12:35:24.373174 2016 tablet_bootstrap.cc:492] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa: Bootstrap starting.
I20260619 12:35:24.373710 2015 tablet_bootstrap.cc:492] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155: Bootstrap starting.
I20260619 12:35:24.375530 2016 tablet_bootstrap.cc:654] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:24.376060 2015 tablet_bootstrap.cc:654] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:24.376345 2016 log.cc:824] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:24.376947 2015 log.cc:824] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:24.377866 2014 tablet_bootstrap.cc:654] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:24.378690 2014 log.cc:824] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:24.379078 2016 tablet_bootstrap.cc:492] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa: No bootstrap required, opened a new log
I20260619 12:35:24.379127 2015 tablet_bootstrap.cc:492] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155: No bootstrap required, opened a new log
I20260619 12:35:24.379317 2016 ts_tablet_manager.cc:1397] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:35:24.379323 2015 ts_tablet_manager.cc:1397] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:35:24.380913 2014 tablet_bootstrap.cc:492] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01: No bootstrap required, opened a new log
I20260619 12:35:24.381242 2014 ts_tablet_manager.cc:1397] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260619 12:35:24.382671 2015 raft_consensus.cc:348] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.382948 2015 raft_consensus.cc:374] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:24.383015 2015 raft_consensus.cc:729] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ae0e0deb484b0daed055910f7c2155, State: Initialized, Role: FOLLOWER
I20260619 12:35:24.383565 2015 consensus_queue.cc:260] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.383620 2016 raft_consensus.cc:348] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.383891 2016 raft_consensus.cc:374] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:24.383972 2016 raft_consensus.cc:729] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb99b9027dd740c687a8e9ffdff6ddfa, State: Initialized, Role: FOLLOWER
I20260619 12:35:24.384393 1991 heartbeater.cc:503] Master 127.0.98.62:33529 was elected leader, sending a full tablet report...
I20260619 12:35:24.384434 2016 consensus_queue.cc:260] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.384804 2015 ts_tablet_manager.cc:1428] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 12:35:24.385459 2016 ts_tablet_manager.cc:1428] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260619 12:35:24.385848 1851 heartbeater.cc:503] Master 127.0.98.62:33529 was elected leader, sending a full tablet report...
I20260619 12:35:24.387233 2014 raft_consensus.cc:348] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.387589 2014 raft_consensus.cc:374] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:24.387643 2014 raft_consensus.cc:729] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b225da14c9844021bcb88e91cfb28b01, State: Initialized, Role: FOLLOWER
I20260619 12:35:24.388095 2014 consensus_queue.cc:260] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.389151 1711 heartbeater.cc:503] Master 127.0.98.62:33529 was elected leader, sending a full tablet report...
I20260619 12:35:24.389209 2014 ts_tablet_manager.cc:1428] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260619 12:35:24.425833 2020 raft_consensus.cc:482] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:24.426088 2020 raft_consensus.cc:504] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.427320 2020 leader_election.cc:288] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b225da14c9844021bcb88e91cfb28b01 (127.0.98.1:46409), eb99b9027dd740c687a8e9ffdff6ddfa (127.0.98.2:36475)
I20260619 12:35:24.431111 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.427827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54781 (local address 127.0.98.1:46409)
0619 12:35:24.427966 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.427971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.427990 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:35:24.428157 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.428160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.428207 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.428308 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:24.428564 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.429072 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.430126 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.430536 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.430653 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.430721 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.430910 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.430979 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 12:35:24.431111 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.427704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46409 (local address 127.0.98.3:54781)
0619 12:35:24.427926 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.427942 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.428045 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.428321 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.428324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.428342 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:24.428458 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.428464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.429185 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.429191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.430005 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.430014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.430669 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.430673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.430788 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.430806 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.430895 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.430979 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260619 12:35:24.431665 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.427710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36475 (local address 127.0.98.3:35577)
0619 12:35:24.428131 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.428145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.428258 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.428491 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.428494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.428509 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:24.428635 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.428640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.429366 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.429372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.430548 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.430557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.431223 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.431226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.431322 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.431344 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.431430 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.431526 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":32,"thread_start_us":129,"threads_started":1}
I20260619 12:35:24.431679 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.427803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35577 (local address 127.0.98.2:36475)
0619 12:35:24.427919 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.427924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.428230 (+ 306us) server_negotiation.cc:407] Connection header received
0619 12:35:24.428342 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.428345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.428396 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.428492 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:24.428751 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.429254 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.430676 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.431092 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.431215 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.431274 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.431448 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.431551 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:35:24.432003 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da6b3bf75684d37b4830f6285adb078" candidate_uuid: "f5ae0e0deb484b0daed055910f7c2155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b225da14c9844021bcb88e91cfb28b01" is_pre_election: true
I20260619 12:35:24.432365 1665 raft_consensus.cc:2454] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ae0e0deb484b0daed055910f7c2155 in term 0.
I20260619 12:35:24.432320 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da6b3bf75684d37b4830f6285adb078" candidate_uuid: "f5ae0e0deb484b0daed055910f7c2155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" is_pre_election: true
I20260619 12:35:24.432624 1805 raft_consensus.cc:2454] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ae0e0deb484b0daed055910f7c2155 in term 0.
I20260619 12:35:24.432909 1876 leader_election.cc:302] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b225da14c9844021bcb88e91cfb28b01, f5ae0e0deb484b0daed055910f7c2155; no voters:
I20260619 12:35:24.433221 2020 raft_consensus.cc:2781] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:24.433317 2020 raft_consensus.cc:482] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:24.433362 2020 raft_consensus.cc:3037] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:24.454941 2020 raft_consensus.cc:504] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.455602 2020 leader_election.cc:288] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [CANDIDATE]: Term 1 election: Requested vote from peers b225da14c9844021bcb88e91cfb28b01 (127.0.98.1:46409), eb99b9027dd740c687a8e9ffdff6ddfa (127.0.98.2:36475)
I20260619 12:35:24.456108 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da6b3bf75684d37b4830f6285adb078" candidate_uuid: "f5ae0e0deb484b0daed055910f7c2155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b225da14c9844021bcb88e91cfb28b01"
I20260619 12:35:24.456115 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da6b3bf75684d37b4830f6285adb078" candidate_uuid: "f5ae0e0deb484b0daed055910f7c2155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa"
I20260619 12:35:24.456280 1665 raft_consensus.cc:3037] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:24.456280 1805 raft_consensus.cc:3037] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:24.457499 1665 raft_consensus.cc:2454] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ae0e0deb484b0daed055910f7c2155 in term 1.
I20260619 12:35:24.457950 1876 leader_election.cc:302] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b225da14c9844021bcb88e91cfb28b01, f5ae0e0deb484b0daed055910f7c2155; no voters:
I20260619 12:35:24.458195 2020 raft_consensus.cc:2781] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:24.458406 1805 raft_consensus.cc:2454] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ae0e0deb484b0daed055910f7c2155 in term 1.
I20260619 12:35:24.458451 2020 raft_consensus.cc:686] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [term 1 LEADER]: Becoming Leader. State: Replica: f5ae0e0deb484b0daed055910f7c2155, State: Running, Role: LEADER
I20260619 12:35:24.458865 2020 consensus_queue.cc:237] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.462663 1507 catalog_manager.cc:5495] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5ae0e0deb484b0daed055910f7c2155 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f5ae0e0deb484b0daed055910f7c2155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } health_report { overall_health: HEALTHY } } }
I20260619 12:35:24.492389 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.488462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.0.98.3:40477)
0619 12:35:24.488591 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.488594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.488609 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:35:24.488787 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.488789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.488849 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.488949 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:24.489272 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.489774 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.490719 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.490948 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.491041 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.491391 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:24.491927 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:24.492034 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.492202 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.492262 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 12:35:24.498876 1665 raft_consensus.cc:1261] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01 [term 1 FOLLOWER]: Refusing update from remote peer f5ae0e0deb484b0daed055910f7c2155: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:24.498876 1805 raft_consensus.cc:1261] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa [term 1 FOLLOWER]: Refusing update from remote peer f5ae0e0deb484b0daed055910f7c2155: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:24.499678 2020 consensus_queue.cc:1035] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:24.500037 2024 consensus_queue.cc:1035] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:24.508132 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298549860338733056 to 7298549860187324416. Current Snapshot: MvccSnapshot[applied={T|T < 7298549860338733056}]
I20260619 12:35:24.519840 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:24.520035 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:24.522435 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb: Sending DeleteTablet for 3 replicas of tablet 6da6b3bf75684d37b4830f6285adb078
I20260619 12:35:24.523337 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 6da6b3bf75684d37b4830f6285adb078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39772
I20260619 12:35:24.523339 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 6da6b3bf75684d37b4830f6285adb078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46560
I20260619 12:35:24.523921 1918 tablet_service.cc:1514] Processing DeleteTablet for tablet 6da6b3bf75684d37b4830f6285adb078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50838
I20260619 12:35:24.524201 2036 tablet_replica.cc:330] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01: stopping tablet replica
I20260619 12:35:24.524334 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:35:24.524741 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:24.524812 2036 raft_consensus.cc:2229] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:24.525219 2036 raft_consensus.cc:2258] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:24.528643 2037 tablet_replica.cc:330] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa: stopping tablet replica
I20260619 12:35:24.529021 2037 raft_consensus.cc:2229] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:24.529366 2037 raft_consensus.cc:2258] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:24.530594 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2221f414134c1e9bc2f79de880c45c (DEFAULT_TABLE table=test-table [id=62e8e6683b3f44af91e60bee10ccabb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:24.530867 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2221f414134c1e9bc2f79de880c45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:24.530908 2036 ts_tablet_manager.cc:1905] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:24.532379 2038 tablet_replica.cc:330] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155: stopping tablet replica
I20260619 12:35:24.532792 2038 raft_consensus.cc:2229] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:35:24.533154 2038 raft_consensus.cc:2258] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:24.533515 2016 tablet_bootstrap.cc:492] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa: Bootstrap starting.
I20260619 12:35:24.533720 1918 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2221f414134c1e9bc2f79de880c45c (DEFAULT_TABLE table=test-table [id=62e8e6683b3f44af91e60bee10ccabb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:24.533988 1918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2221f414134c1e9bc2f79de880c45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:24.534545 2016 tablet_bootstrap.cc:654] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:24.534760 2037 ts_tablet_manager.cc:1905] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:24.536347 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2221f414134c1e9bc2f79de880c45c (DEFAULT_TABLE table=test-table [id=62e8e6683b3f44af91e60bee10ccabb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:24.537282 2038 ts_tablet_manager.cc:1905] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:24.537860 2015 tablet_bootstrap.cc:492] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155: Bootstrap starting.
I20260619 12:35:24.538928 2016 tablet_bootstrap.cc:492] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa: No bootstrap required, opened a new log
I20260619 12:35:24.539026 2015 tablet_bootstrap.cc:654] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:24.539038 2016 ts_tablet_manager.cc:1397] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260619 12:35:24.540040 2016 raft_consensus.cc:348] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.540161 2016 raft_consensus.cc:374] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:24.540194 2016 raft_consensus.cc:729] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb99b9027dd740c687a8e9ffdff6ddfa, State: Initialized, Role: FOLLOWER
I20260619 12:35:24.540324 2016 consensus_queue.cc:260] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.541841 2015 tablet_bootstrap.cc:492] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155: No bootstrap required, opened a new log
I20260619 12:35:24.541939 2015 ts_tablet_manager.cc:1397] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260619 12:35:24.542332 2016 ts_tablet_manager.cc:1428] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:35:24.542428 2015 raft_consensus.cc:348] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.542549 2015 raft_consensus.cc:374] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:24.542580 2015 raft_consensus.cc:729] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ae0e0deb484b0daed055910f7c2155, State: Initialized, Role: FOLLOWER
I20260619 12:35:24.542697 2015 consensus_queue.cc:260] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.542934 2015 ts_tablet_manager.cc:1428] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:35:24.544538 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2221f414134c1e9bc2f79de880c45c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260619 12:35:24.550369 1992 tablet.cc:2369] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:35:24.550999 2014 tablet_bootstrap.cc:492] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01: Bootstrap starting.
I20260619 12:35:24.552065 2014 tablet_bootstrap.cc:654] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:24.552650 2021 raft_consensus.cc:482] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:24.552760 2021 raft_consensus.cc:504] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.553877 2021 leader_election.cc:288] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b225da14c9844021bcb88e91cfb28b01 (127.0.98.1:46409), f5ae0e0deb484b0daed055910f7c2155 (127.0.98.3:40477)
I20260619 12:35:24.554700 2014 tablet_bootstrap.cc:492] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01: No bootstrap required, opened a new log
I20260619 12:35:24.554797 2014 ts_tablet_manager.cc:1397] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 12:35:24.555778 2014 raft_consensus.cc:348] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.555931 2014 raft_consensus.cc:374] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:24.555987 2014 raft_consensus.cc:729] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b225da14c9844021bcb88e91cfb28b01, State: Initialized, Role: FOLLOWER
I20260619 12:35:24.556306 2014 consensus_queue.cc:260] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.556474 2022 raft_consensus.cc:482] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:24.556561 2022 raft_consensus.cc:504] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.557745 2022 leader_election.cc:288] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb99b9027dd740c687a8e9ffdff6ddfa (127.0.98.2:36475), f5ae0e0deb484b0daed055910f7c2155 (127.0.98.3:40477)
I20260619 12:35:24.559185 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.554147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40477 (local address 127.0.98.2:43393)
0619 12:35:24.554268 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.554288 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.554377 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.555094 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.555098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.555114 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:24.555250 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.555255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.556337 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.556341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.557657 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.557668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.558656 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.558660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.558853 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.558876 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.558950 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.559027 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":40}
I20260619 12:35:24.559405 2014 ts_tablet_manager.cc:1428] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 12:35:24.559695 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.554598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43393 (local address 127.0.98.3:40477)
0619 12:35:24.554719 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.554725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.554742 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:35:24.554794 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.554797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.554860 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.554947 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:24.555380 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.556197 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.557804 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.558424 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.558527 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.558596 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.559496 (+ 900us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.559579 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 12:35:24.560235 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2221f414134c1e9bc2f79de880c45c" candidate_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ae0e0deb484b0daed055910f7c2155" is_pre_election: true
I20260619 12:35:24.560491 1945 raft_consensus.cc:2454] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb99b9027dd740c687a8e9ffdff6ddfa in term 0.
I20260619 12:35:24.561045 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.554560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50823 (local address 127.0.98.1:46409)
0619 12:35:24.554685 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.554690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.555886 (+ 1196us) server_negotiation.cc:407] Connection header received
0619 12:35:24.556226 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.556228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.556272 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.556319 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:24.556920 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.557629 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.559506 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.560063 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.560139 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.560208 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.560851 (+ 643us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.560932 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 12:35:24.561370 1738 leader_election.cc:302] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eb99b9027dd740c687a8e9ffdff6ddfa, f5ae0e0deb484b0daed055910f7c2155; no voters:
I20260619 12:35:24.561628 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.554184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46409 (local address 127.0.98.2:50823)
0619 12:35:24.555803 (+ 1619us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.555821 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.556103 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.556593 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.556596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.556614 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:24.556795 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.556801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.557773 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.557776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.559148 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.559161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.560512 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.560515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.560692 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.560717 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.561304 (+ 587us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.561397 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1533,"mutex_wait_us":46,"thread_start_us":82,"threads_started":1}
I20260619 12:35:24.562155 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2221f414134c1e9bc2f79de880c45c" candidate_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b225da14c9844021bcb88e91cfb28b01" is_pre_election: true
I20260619 12:35:24.562322 1665 raft_consensus.cc:2454] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb99b9027dd740c687a8e9ffdff6ddfa in term 0.
I20260619 12:35:24.562340 2021 raft_consensus.cc:2781] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:24.562409 2021 raft_consensus.cc:482] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:24.562441 2021 raft_consensus.cc:3037] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:24.568041 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.558788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42093 (local address 127.0.98.2:36475)
0619 12:35:24.558894 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.558898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.563875 (+ 4977us) server_negotiation.cc:407] Connection header received
0619 12:35:24.564027 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.564030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.564080 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.564148 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:24.564544 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.565109 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.566698 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.567146 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.567268 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.567337 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.567801 (+ 464us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.567897 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 12:35:24.568157 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.559072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38447 (local address 127.0.98.3:40477)
0619 12:35:24.559840 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.559845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.564230 (+ 4385us) server_negotiation.cc:407] Connection header received
0619 12:35:24.564316 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.564318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.564365 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.564455 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:24.564710 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.565444 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.566815 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.567461 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.567589 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.567649 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.567951 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.568020 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":73,"threads_started":1}
I20260619 12:35:24.568612 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.558105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36475 (local address 127.0.98.1:42093)
0619 12:35:24.563776 (+ 5671us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.563789 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.563916 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.564262 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.564267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.564286 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:24.564405 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.564410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.565220 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.565223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.566552 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.566562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.567477 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.567505 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.567648 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.567687 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.568322 (+ 635us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.568405 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5614,"thread_start_us":87,"threads_started":1}
I20260619 12:35:24.568822 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.558479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40477 (local address 127.0.98.1:38447)
0619 12:35:24.564141 (+ 5662us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.564154 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.564234 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.564483 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.564485 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.564505 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:24.564607 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.564613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.565564 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.565567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.566701 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.566709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.567710 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.567714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.567835 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.567857 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.567934 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.568005 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5584,"thread_start_us":88,"threads_started":1}
I20260619 12:35:24.569396 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2221f414134c1e9bc2f79de880c45c" candidate_uuid: "b225da14c9844021bcb88e91cfb28b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ae0e0deb484b0daed055910f7c2155" is_pre_election: true
I20260619 12:35:24.569402 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2221f414134c1e9bc2f79de880c45c" candidate_uuid: "b225da14c9844021bcb88e91cfb28b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" is_pre_election: true
I20260619 12:35:24.569564 1945 raft_consensus.cc:2454] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b225da14c9844021bcb88e91cfb28b01 in term 0.
I20260619 12:35:24.569937 1598 leader_election.cc:302] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b225da14c9844021bcb88e91cfb28b01, f5ae0e0deb484b0daed055910f7c2155; no voters:
I20260619 12:35:24.570202 2022 raft_consensus.cc:2781] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:24.570268 2022 raft_consensus.cc:482] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:24.570333 2022 raft_consensus.cc:3037] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Advancing to term 1
W20260619 12:35:24.574282 1852 tablet.cc:2369] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:35:24.589591 1712 tablet.cc:2369] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:35:24.624238 2036 ts_tablet_manager.cc:1918] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:24.624238 2037 ts_tablet_manager.cc:1918] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:24.624392 2037 log.cc:1196] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/6da6b3bf75684d37b4830f6285adb078
I20260619 12:35:24.624392 2036 log.cc:1196] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/wal/wals/6da6b3bf75684d37b4830f6285adb078
I20260619 12:35:24.624785 2036 ts_tablet_manager.cc:1939] T 6da6b3bf75684d37b4830f6285adb078 P b225da14c9844021bcb88e91cfb28b01: Deleting consensus metadata
I20260619 12:35:24.624764 2037 ts_tablet_manager.cc:1939] T 6da6b3bf75684d37b4830f6285adb078 P eb99b9027dd740c687a8e9ffdff6ddfa: Deleting consensus metadata
I20260619 12:35:24.625054 2021 raft_consensus.cc:504] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.625054 2022 raft_consensus.cc:504] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.625506 2022 leader_election.cc:288] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [CANDIDATE]: Term 1 election: Requested vote from peers eb99b9027dd740c687a8e9ffdff6ddfa (127.0.98.2:36475), f5ae0e0deb484b0daed055910f7c2155 (127.0.98.3:40477)
I20260619 12:35:24.625538 2021 leader_election.cc:288] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [CANDIDATE]: Term 1 election: Requested vote from peers b225da14c9844021bcb88e91cfb28b01 (127.0.98.1:46409), f5ae0e0deb484b0daed055910f7c2155 (127.0.98.3:40477)
I20260619 12:35:24.626078 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2221f414134c1e9bc2f79de880c45c" candidate_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b225da14c9844021bcb88e91cfb28b01"
I20260619 12:35:24.626272 1665 raft_consensus.cc:2379] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eb99b9027dd740c687a8e9ffdff6ddfa in current term 1: Already voted for candidate b225da14c9844021bcb88e91cfb28b01 in this term.
I20260619 12:35:24.626668 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2221f414134c1e9bc2f79de880c45c" candidate_uuid: "b225da14c9844021bcb88e91cfb28b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa"
I20260619 12:35:24.627797 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2221f414134c1e9bc2f79de880c45c" candidate_uuid: "b225da14c9844021bcb88e91cfb28b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ae0e0deb484b0daed055910f7c2155"
I20260619 12:35:24.627821 1492 catalog_manager.cc:4841] TS b225da14c9844021bcb88e91cfb28b01 (127.0.98.1:46409): tablet 6da6b3bf75684d37b4830f6285adb078 (table test-table [id=d58bacf450c543f49413aaf388c0df98]) successfully deleted
I20260619 12:35:24.627936 1944 raft_consensus.cc:3037] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:24.629271 1805 raft_consensus.cc:2379] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b225da14c9844021bcb88e91cfb28b01 in current term 1: Already voted for candidate eb99b9027dd740c687a8e9ffdff6ddfa in this term.
I20260619 12:35:24.629263 1944 raft_consensus.cc:2454] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b225da14c9844021bcb88e91cfb28b01 in term 1.
I20260619 12:35:24.629632 1493 catalog_manager.cc:4841] TS eb99b9027dd740c687a8e9ffdff6ddfa (127.0.98.2:36475): tablet 6da6b3bf75684d37b4830f6285adb078 (table test-table [id=d58bacf450c543f49413aaf388c0df98]) successfully deleted
I20260619 12:35:24.629901 1598 leader_election.cc:302] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b225da14c9844021bcb88e91cfb28b01, f5ae0e0deb484b0daed055910f7c2155; no voters: eb99b9027dd740c687a8e9ffdff6ddfa
I20260619 12:35:24.630055 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2221f414134c1e9bc2f79de880c45c" candidate_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ae0e0deb484b0daed055910f7c2155"
I20260619 12:35:24.630203 2022 raft_consensus.cc:2781] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:24.630240 1945 raft_consensus.cc:2379] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eb99b9027dd740c687a8e9ffdff6ddfa in current term 1: Already voted for candidate b225da14c9844021bcb88e91cfb28b01 in this term.
I20260619 12:35:24.630424 2022 raft_consensus.cc:686] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [term 1 LEADER]: Becoming Leader. State: Replica: b225da14c9844021bcb88e91cfb28b01, State: Running, Role: LEADER
I20260619 12:35:24.630674 2022 consensus_queue.cc:237] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.631168 1738 leader_election.cc:302] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb99b9027dd740c687a8e9ffdff6ddfa; no voters: b225da14c9844021bcb88e91cfb28b01, f5ae0e0deb484b0daed055910f7c2155
I20260619 12:35:24.631218 2038 ts_tablet_manager.cc:1918] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:24.631316 2038 log.cc:1196] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/6da6b3bf75684d37b4830f6285adb078
I20260619 12:35:24.631450 2021 raft_consensus.cc:2726] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 12:35:24.631721 2038 ts_tablet_manager.cc:1939] T 6da6b3bf75684d37b4830f6285adb078 P f5ae0e0deb484b0daed055910f7c2155: Deleting consensus metadata
I20260619 12:35:24.633325 1507 catalog_manager.cc:5495] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 reported cstate change: term changed from 0 to 1, leader changed from <none> to b225da14c9844021bcb88e91cfb28b01 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b225da14c9844021bcb88e91cfb28b01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } health_report { overall_health: UNKNOWN } } }
Jun 19 12:35:24 dist-test-slave-kdrb krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:24.679314 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.669404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33529 (local address 127.0.0.1:34424)
0619 12:35:24.669747 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.669764 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.669877 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.670137 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.670139 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.670529 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:24.670625 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.670631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.671705 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.671708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.672330 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.672340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.672944 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.672946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.673011 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.673773 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:24.673792 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:24.675368 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:24.677430 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:24.677440 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:24.677448 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:24.677792 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:24.678082 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:24.678086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:24.678088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:24.678187 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:24.678656 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:24.678665 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:24.678819 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.678950 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.679125 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":145,"threads_started":1}
I20260619 12:35:24.679323 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.669513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.0.98.62:33529)
0619 12:35:24.669649 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.669654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.669845 (+ 191us) server_negotiation.cc:407] Connection header received
0619 12:35:24.669959 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.669962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.670009 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.670107 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:24.670747 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.671587 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.672457 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.672816 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.672916 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.675581 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:24.675585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:24.675588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:24.675616 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:24.677271 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:24.677898 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:24.677901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:24.677902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:24.677985 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:24.678282 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:24.678287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:24.678293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:24.678534 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:24.678623 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.679012 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.679205 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":46}
I20260619 12:35:24.684185 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34424:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 12:35:24.685125 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:24.690519 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3681d53f2340b0a5001b90c3b67289 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d041d83d7b394f6d8c3bb904ec19a96b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:35:24.690630 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3681d53f2340b0a5001b90c3b67289 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d041d83d7b394f6d8c3bb904ec19a96b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:35:24.690970 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3681d53f2340b0a5001b90c3b67289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:24.691000 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3681d53f2340b0a5001b90c3b67289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:24.694150 2014 tablet_bootstrap.cc:492] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01: Bootstrap starting.
I20260619 12:35:24.694386 1918 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3681d53f2340b0a5001b90c3b67289 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d041d83d7b394f6d8c3bb904ec19a96b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:35:24.694772 1918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3681d53f2340b0a5001b90c3b67289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:24.695125 2014 tablet_bootstrap.cc:654] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:24.696480 2014 tablet_bootstrap.cc:492] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01: No bootstrap required, opened a new log
I20260619 12:35:24.696573 2014 ts_tablet_manager.cc:1397] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:35:24.697036 2014 raft_consensus.cc:348] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.697122 2016 tablet_bootstrap.cc:492] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa: Bootstrap starting.
I20260619 12:35:24.697166 2014 raft_consensus.cc:374] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:24.697204 2014 raft_consensus.cc:729] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b225da14c9844021bcb88e91cfb28b01, State: Initialized, Role: FOLLOWER
I20260619 12:35:24.697325 2014 consensus_queue.cc:260] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.697480 2015 tablet_bootstrap.cc:492] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155: Bootstrap starting.
I20260619 12:35:24.697583 2022 tablet_replica.cc:439] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } } }
I20260619 12:35:24.697717 2022 tablet_replica.cc:442] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:24.697928 2014 ts_tablet_manager.cc:1428] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:35:24.698132 2016 tablet_bootstrap.cc:654] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:24.698348 2015 tablet_bootstrap.cc:654] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:24.699384 2016 tablet_bootstrap.cc:492] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa: No bootstrap required, opened a new log
I20260619 12:35:24.699499 2015 tablet_bootstrap.cc:492] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155: No bootstrap required, opened a new log
I20260619 12:35:24.699499 2016 ts_tablet_manager.cc:1397] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:35:24.699566 2015 ts_tablet_manager.cc:1397] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 12:35:24.699951 2015 raft_consensus.cc:348] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.699973 2016 raft_consensus.cc:348] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.700075 2016 raft_consensus.cc:374] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:24.700075 2015 raft_consensus.cc:374] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:24.700115 2015 raft_consensus.cc:729] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ae0e0deb484b0daed055910f7c2155, State: Initialized, Role: FOLLOWER
I20260619 12:35:24.700115 2016 raft_consensus.cc:729] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb99b9027dd740c687a8e9ffdff6ddfa, State: Initialized, Role: FOLLOWER
I20260619 12:35:24.700243 2016 consensus_queue.cc:260] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.700243 2015 consensus_queue.cc:260] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.700515 2015 ts_tablet_manager.cc:1428] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 12:35:24.700516 2016 ts_tablet_manager.cc:1428] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 12:35:24.700491 2021 tablet_replica.cc:439] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } } }
I20260619 12:35:24.700649 2021 tablet_replica.cc:442] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:24.701146 2020 tablet_replica.cc:439] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } } }
I20260619 12:35:24.701287 2020 tablet_replica.cc:442] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:24.709438 1494 catalog_manager.cc:4841] TS f5ae0e0deb484b0daed055910f7c2155 (127.0.98.3:40477): tablet 6da6b3bf75684d37b4830f6285adb078 (table test-table [id=d58bacf450c543f49413aaf388c0df98]) successfully deleted
I20260619 12:35:24.887420 2020 raft_consensus.cc:482] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:24.887610 2020 raft_consensus.cc:504] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.888087 2020 leader_election.cc:288] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b225da14c9844021bcb88e91cfb28b01 (127.0.98.1:46409), eb99b9027dd740c687a8e9ffdff6ddfa (127.0.98.2:36475)
I20260619 12:35:24.888546 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3681d53f2340b0a5001b90c3b67289" candidate_uuid: "f5ae0e0deb484b0daed055910f7c2155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" is_pre_election: true
I20260619 12:35:24.888548 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3681d53f2340b0a5001b90c3b67289" candidate_uuid: "f5ae0e0deb484b0daed055910f7c2155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b225da14c9844021bcb88e91cfb28b01" is_pre_election: true
I20260619 12:35:24.888720 1805 raft_consensus.cc:2454] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ae0e0deb484b0daed055910f7c2155 in term 0.
I20260619 12:35:24.888736 1665 raft_consensus.cc:2454] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ae0e0deb484b0daed055910f7c2155 in term 0.
I20260619 12:35:24.889101 1877 leader_election.cc:302] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b225da14c9844021bcb88e91cfb28b01, eb99b9027dd740c687a8e9ffdff6ddfa, f5ae0e0deb484b0daed055910f7c2155; no voters:
I20260619 12:35:24.889305 2020 raft_consensus.cc:2781] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:24.889365 2020 raft_consensus.cc:482] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:24.889410 2020 raft_consensus.cc:3037] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:24.890369 2020 raft_consensus.cc:504] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.890744 2020 leader_election.cc:288] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [CANDIDATE]: Term 1 election: Requested vote from peers b225da14c9844021bcb88e91cfb28b01 (127.0.98.1:46409), eb99b9027dd740c687a8e9ffdff6ddfa (127.0.98.2:36475)
I20260619 12:35:24.891156 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3681d53f2340b0a5001b90c3b67289" candidate_uuid: "f5ae0e0deb484b0daed055910f7c2155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b225da14c9844021bcb88e91cfb28b01"
I20260619 12:35:24.891273 1665 raft_consensus.cc:3037] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:24.891228 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3681d53f2340b0a5001b90c3b67289" candidate_uuid: "f5ae0e0deb484b0daed055910f7c2155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa"
I20260619 12:35:24.891399 1805 raft_consensus.cc:3037] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:24.892354 1665 raft_consensus.cc:2454] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ae0e0deb484b0daed055910f7c2155 in term 1.
I20260619 12:35:24.892443 1805 raft_consensus.cc:2454] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ae0e0deb484b0daed055910f7c2155 in term 1.
I20260619 12:35:24.892771 1876 leader_election.cc:302] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b225da14c9844021bcb88e91cfb28b01, f5ae0e0deb484b0daed055910f7c2155; no voters:
I20260619 12:35:24.892997 2020 raft_consensus.cc:2781] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:24.893105 2020 raft_consensus.cc:686] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [term 1 LEADER]: Becoming Leader. State: Replica: f5ae0e0deb484b0daed055910f7c2155, State: Running, Role: LEADER
I20260619 12:35:24.893245 2020 consensus_queue.cc:237] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } }
I20260619 12:35:24.893966 2024 tablet_replica.cc:439] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155: TxnStatusTablet state changed. Reason: New leader f5ae0e0deb484b0daed055910f7c2155. Latest consensus state: current_term: 1 leader_uuid: "f5ae0e0deb484b0daed055910f7c2155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } } }
I20260619 12:35:24.894073 2024 tablet_replica.cc:442] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:35:24.894315 2058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 12:35:24.894498 2058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 12:35:24.895124 1508 catalog_manager.cc:5495] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5ae0e0deb484b0daed055910f7c2155 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f5ae0e0deb484b0daed055910f7c2155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } health_report { overall_health: HEALTHY } } }
I20260619 12:35:24.939222 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.935720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.0.98.3:40477)
0619 12:35:24.935871 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.935876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.935894 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:35:24.936025 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.936028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.936083 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.936177 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:24.936379 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.937180 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.938027 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.938329 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.938433 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.938696 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:24.938784 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:24.938853 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.939040 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.939092 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 12:35:24.939222 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.935634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40477 (local address 127.0.0.1:50846)
0619 12:35:24.935774 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.935798 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.935906 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.936186 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.936188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.936194 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:35:24.936286 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.936290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.937300 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.937303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.937909 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.937918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.938439 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.938442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.938520 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.938567 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:35:24.938862 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:35:24.938872 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.938970 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.939071 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 12:35:24.942195 1805 raft_consensus.cc:1261] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa [term 1 FOLLOWER]: Refusing update from remote peer f5ae0e0deb484b0daed055910f7c2155: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:24.942195 1665 raft_consensus.cc:1261] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01 [term 1 FOLLOWER]: Refusing update from remote peer f5ae0e0deb484b0daed055910f7c2155: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:24.942624 2020 consensus_queue.cc:1035] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:24.942797 2024 consensus_queue.cc:1035] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:24.944094 2021 tablet_replica.cc:439] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa: TxnStatusTablet state changed. Reason: New leader f5ae0e0deb484b0daed055910f7c2155. Latest consensus state: current_term: 1 leader_uuid: "f5ae0e0deb484b0daed055910f7c2155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } } }
I20260619 12:35:24.944216 2021 tablet_replica.cc:442] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:24.945205 2022 tablet_replica.cc:439] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01: TxnStatusTablet state changed. Reason: New leader f5ae0e0deb484b0daed055910f7c2155. Latest consensus state: current_term: 1 leader_uuid: "f5ae0e0deb484b0daed055910f7c2155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } } }
I20260619 12:35:24.945324 2022 tablet_replica.cc:442] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:24.945367 2020 tablet_replica.cc:439] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5ae0e0deb484b0daed055910f7c2155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } } }
I20260619 12:35:24.945492 2020 tablet_replica.cc:442] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:35:24.945596 2027 mvcc.cc:205] Tried to move back new op lower bound from 7298549862158610432 to 7298549861964296192. Current Snapshot: MvccSnapshot[applied={T|T < 7298549862158610432}]
I20260619 12:35:24.946851 2024 tablet_replica.cc:439] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5ae0e0deb484b0daed055910f7c2155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } } }
I20260619 12:35:24.947021 2024 tablet_replica.cc:442] T 9e3681d53f2340b0a5001b90c3b67289 P f5ae0e0deb484b0daed055910f7c2155: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:35:24.947335 2022 tablet_replica.cc:439] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5ae0e0deb484b0daed055910f7c2155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } } }
I20260619 12:35:24.947438 2022 tablet_replica.cc:442] T 9e3681d53f2340b0a5001b90c3b67289 P b225da14c9844021bcb88e91cfb28b01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:24.947801 2021 tablet_replica.cc:439] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5ae0e0deb484b0daed055910f7c2155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b225da14c9844021bcb88e91cfb28b01" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46409 } } peers { permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 } } }
I20260619 12:35:24.947908 2021 tablet_replica.cc:442] T 9e3681d53f2340b0a5001b90c3b67289 P eb99b9027dd740c687a8e9ffdff6ddfa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:24.954994 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.951283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.0.98.1:46409)
0619 12:35:24.951728 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.951731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.951741 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:35:24.951799 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.951801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.951848 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.951970 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:24.952192 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.952710 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.953521 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.953754 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.953862 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.954122 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:24.954590 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:24.954713 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.954784 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.954841 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379}
I20260619 12:35:24.967366 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.963563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41467 (local address 127.0.98.3:40477)
0619 12:35:24.963719 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.963725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.963944 (+ 219us) server_negotiation.cc:407] Connection header received
0619 12:35:24.964066 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.964068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.964114 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.964176 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:24.964441 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.964942 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.966221 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.966428 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.966511 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.966871 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:24.966948 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:24.967009 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.967196 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.967250 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:35:24.967406 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.963437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40477 (local address 127.0.98.1:41467)
0619 12:35:24.963829 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.963846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.963977 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.964207 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.964210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.964217 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:35:24.964339 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.964344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.965049 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.965052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.966098 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.966111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.966547 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.966550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.966648 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.966733 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:35:24.967038 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:35:24.967045 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.967112 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.967200 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":182,"threads_started":1}
I20260619 12:35:24.975215 1805 raft_consensus.cc:1261] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [term 1 FOLLOWER]: Refusing update from remote peer b225da14c9844021bcb88e91cfb28b01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:24.975266 1945 raft_consensus.cc:1261] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155 [term 1 FOLLOWER]: Refusing update from remote peer b225da14c9844021bcb88e91cfb28b01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:24.975755 2022 consensus_queue.cc:1035] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5ae0e0deb484b0daed055910f7c2155" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:24.975996 2046 consensus_queue.cc:1035] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb99b9027dd740c687a8e9ffdff6ddfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:24.981066 2028 mvcc.cc:205] Tried to move back new op lower bound from 7298549862293831680 to 7298549860890087424. Current Snapshot: MvccSnapshot[applied={T|T < 7298549862293831680}]
I20260619 12:35:24.998600 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.994706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42147 (local address 127.0.98.1:46409)
0619 12:35:24.994825 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:24.994831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:24.995044 (+ 213us) server_negotiation.cc:407] Connection header received
0619 12:35:24.995163 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:24.995166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:24.995207 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:24.995269 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:24.995560 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.996248 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.997531 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.997764 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.997875 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.998156 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:24.998239 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:24.998316 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:24.998407 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 12:35:24.998458 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 12:35:24.998611 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:24.994570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46409 (local address 127.0.98.3:42147)
0619 12:35:24.994951 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:24.994968 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:24.995078 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:24.995307 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:24.995310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:24.995317 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:35:24.995427 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.995433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.996574 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.996576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.997397 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:24.997408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:24.997884 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:24.997887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:24.997977 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:24.998023 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:35:24.998331 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:35:24.998338 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 12:35:24.998398 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 12:35:24.998486 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":143,"threads_started":1}
I20260619 12:35:25.027455 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:25.027674 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:25.030205 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f459e71a25b46118371b7fb144ed5bb: Sending DeleteTablet for 3 replicas of tablet 4b2221f414134c1e9bc2f79de880c45c
I20260619 12:35:25.031059 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b2221f414134c1e9bc2f79de880c45c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46560
I20260619 12:35:25.031118 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b2221f414134c1e9bc2f79de880c45c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39772
I20260619 12:35:25.031251 2036 tablet_replica.cc:330] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01: stopping tablet replica
I20260619 12:35:25.031308 2037 tablet_replica.cc:330] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa: stopping tablet replica
I20260619 12:35:25.031358 2036 raft_consensus.cc:2229] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:35:25.031417 2037 raft_consensus.cc:2229] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:25.031589 2037 raft_consensus.cc:2258] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:25.031662 2036 raft_consensus.cc:2258] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:25.032351 2037 ts_tablet_manager.cc:1905] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:25.033077 2036 ts_tablet_manager.cc:1905] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:25.035377 2037 ts_tablet_manager.cc:1918] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:35:25.035449 2037 log.cc:1196] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/4b2221f414134c1e9bc2f79de880c45c
I20260619 12:35:25.035629 1918 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b2221f414134c1e9bc2f79de880c45c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50838
I20260619 12:35:25.035737 2037 ts_tablet_manager.cc:1939] T 4b2221f414134c1e9bc2f79de880c45c P eb99b9027dd740c687a8e9ffdff6ddfa: Deleting consensus metadata
I20260619 12:35:25.035820 2038 tablet_replica.cc:330] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155: stopping tablet replica
I20260619 12:35:25.035933 2038 raft_consensus.cc:2229] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:25.036037 2038 raft_consensus.cc:2258] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:25.036522 1493 catalog_manager.cc:4841] TS eb99b9027dd740c687a8e9ffdff6ddfa (127.0.98.2:36475): tablet 4b2221f414134c1e9bc2f79de880c45c (table test-table [id=62e8e6683b3f44af91e60bee10ccabb0]) successfully deleted
I20260619 12:35:25.036788 2038 ts_tablet_manager.cc:1905] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:25.039266 2038 ts_tablet_manager.cc:1918] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:35:25.039342 2038 log.cc:1196] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/4b2221f414134c1e9bc2f79de880c45c
I20260619 12:35:25.039775 2038 ts_tablet_manager.cc:1939] T 4b2221f414134c1e9bc2f79de880c45c P f5ae0e0deb484b0daed055910f7c2155: Deleting consensus metadata
I20260619 12:35:25.040561 1494 catalog_manager.cc:4841] TS f5ae0e0deb484b0daed055910f7c2155 (127.0.98.3:40477): tablet 4b2221f414134c1e9bc2f79de880c45c (table test-table [id=62e8e6683b3f44af91e60bee10ccabb0]) successfully deleted
I20260619 12:35:25.040639 2036 ts_tablet_manager.cc:1918] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:35:25.040823 2036 log.cc:1196] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872516972220-392-0/minicluster-data/ts-0/wal/wals/4b2221f414134c1e9bc2f79de880c45c
I20260619 12:35:25.041172 2036 ts_tablet_manager.cc:1939] T 4b2221f414134c1e9bc2f79de880c45c P b225da14c9844021bcb88e91cfb28b01: Deleting consensus metadata
I20260619 12:35:25.042402 1492 catalog_manager.cc:4841] TS b225da14c9844021bcb88e91cfb28b01 (127.0.98.1:46409): tablet 4b2221f414134c1e9bc2f79de880c45c (table test-table [id=62e8e6683b3f44af91e60bee10ccabb0]) successfully deleted
Jun 19 12:35:25 dist-test-slave-kdrb krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872524, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:35:25.054697 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:25.041151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46574 (local address 127.0.98.1:46409)
0619 12:35:25.041281 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:25.041285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:25.042493 (+ 1208us) server_negotiation.cc:407] Connection header received
0619 12:35:25.042749 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:25.042753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:25.042814 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:25.042909 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:25.043942 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:25.044730 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:25.045925 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:25.046264 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:25.046366 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:25.050178 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:25.050188 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:25.050208 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:25.050238 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:25.052745 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:25.053259 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:25.053266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:25.053268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:25.053374 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:25.053638 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:25.053642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:25.053643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:25.053958 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:25.054178 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:25.054385 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 12:35:25.054511 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 12:35:25.056098 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46574
I20260619 12:35:25.066732 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:25.058108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34428 (local address 127.0.98.62:33529)
0619 12:35:25.058230 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:25.058235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:25.058625 (+ 390us) server_negotiation.cc:407] Connection header received
0619 12:35:25.058738 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:25.058742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:25.058795 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:25.058876 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:25.059740 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:25.060544 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:25.061670 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:25.061887 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:25.061976 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:25.063649 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:25.063652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:25.063655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:25.063679 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:25.065119 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:25.065595 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:25.065598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:25.065599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:25.065674 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:25.065951 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:25.065953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:25.065954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:25.066121 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:25.066215 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:25.066475 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 12:35:25.066569 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 12:35:25.067178 1508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34428
I20260619 12:35:25.068749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 1580
I20260619 12:35:25.076025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 1720
I20260619 12:35:25.082077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 1860
I20260619 12:35:25.088937 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 1476
2026-06-19T12:35:25Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2933 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:35:25 dist-test-slave-kdrb krb5kdc[2080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:35:25 dist-test-slave-kdrb krb5kdc[2080](info): set up 2 sockets
Jun 19 12:35:25 dist-test-slave-kdrb krb5kdc[2080](info): commencing operation
krb5kdc: starting...
W20260619 12:35:26.123500 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:35:26 dist-test-slave-kdrb krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:35:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:35:26.247184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36041
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:26.365121 2093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:26.365476 2093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:26.365561 2093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:26.372880 2093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:26.372972 2093 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 12:35:26.372997 2093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:26.373024 2093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:26.373059 2093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:26.373097 2093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:26.382978 2093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36041
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:26.384580 2093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:26.386152 2093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:26.398036 2098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:26.398415 2099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:26.398804 2101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:26.399507 2093 server_base.cc:1034] running on GCE node
I20260619 12:35:26.400260 2093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:26.401697 2093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:26.402971 2093 hybrid_clock.cc:648] HybridClock initialized: now 1781872526402889 us; error 106 us; skew 500 ppm
Jun 19 12:35:26 dist-test-slave-kdrb krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:26.410132 2093 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:35:26.411095 2093 webserver.cc:460] Webserver started at http://127.0.98.62:41411/ using document root <none> and password file <none>
I20260619 12:35:26.411715 2093 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:26.411778 2093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:26.411945 2093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:26.414031 2093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "04d409edecf7498b913732ddc59f44bc"
format_stamp: "Formatted at 2026-06-19 12:35:26 on dist-test-slave-kdrb"
server_key: "12b569f589ed4ecf33afe05b90b26257"
server_key_iv: "83c7a2fd6d0d1d60b1806e4f48e4aa70"
server_key_version: "encryptionkey@0"
I20260619 12:35:26.414544 2093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "04d409edecf7498b913732ddc59f44bc"
format_stamp: "Formatted at 2026-06-19 12:35:26 on dist-test-slave-kdrb"
server_key: "12b569f589ed4ecf33afe05b90b26257"
server_key_iv: "83c7a2fd6d0d1d60b1806e4f48e4aa70"
server_key_version: "encryptionkey@0"
I20260619 12:35:26.418587 2093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:35:26.421269 2108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:26.422596 2093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:35:26.422742 2093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "04d409edecf7498b913732ddc59f44bc"
format_stamp: "Formatted at 2026-06-19 12:35:26 on dist-test-slave-kdrb"
server_key: "12b569f589ed4ecf33afe05b90b26257"
server_key_iv: "83c7a2fd6d0d1d60b1806e4f48e4aa70"
server_key_version: "encryptionkey@0"
I20260619 12:35:26.422863 2093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:26.442301 2093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:26.445863 2093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:26.446092 2093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:26.454309 2093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36041
I20260619 12:35:26.454329 2170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36041 every 8 connection(s)
I20260619 12:35:26.455286 2093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:26.458155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 2093
I20260619 12:35:26.458280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:26.458315 2171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:26.458575 392 external_mini_cluster.cc:1428] Setting key 389f43dfa3c764e51985ca71ba98487d
I20260619 12:35:26.464979 2171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc: Bootstrap starting.
Jun 19 12:35:26 dist-test-slave-kdrb krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:26.467584 2171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:26.468467 2171 log.cc:824] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:26.471239 2171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc: No bootstrap required, opened a new log
I20260619 12:35:26.474627 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:26.460047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.0.98.62:36041)
0619 12:35:26.460487 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:26.460502 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:26.460537 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:35:26.461185 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:26.461202 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:26.461527 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:26.461874 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:26.462652 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.463698 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.464551 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.465032 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.465212 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:26.468053 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:26.468058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:26.468074 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:26.468105 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:26.471022 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:26.471644 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:26.471652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:26.471660 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:26.471831 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:26.472167 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:26.472170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:26.472172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:26.472713 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:26.472927 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:26.473230 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 12:35:26.473560 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":143,"threads_started":1}
I20260619 12:35:26.475550 2171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d409edecf7498b913732ddc59f44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36041 } }
I20260619 12:35:26.475857 2171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:26.475922 2171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04d409edecf7498b913732ddc59f44bc, State: Initialized, Role: FOLLOWER
I20260619 12:35:26.476414 2171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d409edecf7498b913732ddc59f44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36041 } }
I20260619 12:35:26.476545 2171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:26.476626 2171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:26.476727 2171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:26.477909 2171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d409edecf7498b913732ddc59f44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36041 } }
I20260619 12:35:26.478282 2171 leader_election.cc:302] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04d409edecf7498b913732ddc59f44bc; no voters:
I20260619 12:35:26.478647 2171 leader_election.cc:288] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:26.478796 2176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:26.479031 2176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [term 1 LEADER]: Becoming Leader. State: Replica: 04d409edecf7498b913732ddc59f44bc, State: Running, Role: LEADER
I20260619 12:35:26.479395 2176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d409edecf7498b913732ddc59f44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36041 } }
I20260619 12:35:26.480260 2171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:26.481349 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04d409edecf7498b913732ddc59f44bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d409edecf7498b913732ddc59f44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36041 } } }
I20260619 12:35:26.481541 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:26.481352 2178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04d409edecf7498b913732ddc59f44bc. Latest consensus state: current_term: 1 leader_uuid: "04d409edecf7498b913732ddc59f44bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d409edecf7498b913732ddc59f44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36041 } } }
I20260619 12:35:26.481802 2178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:26.482165 2185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:26.485424 2185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:26.491061 2185 catalog_manager.cc:1347] Generated new cluster ID: d6c64389195740368d8ffc65cae69107
I20260619 12:35:26.491155 2185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:26.500098 2185 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:26.501355 2185 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:26.508219 2185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc: Generated new TSK 0
I20260619 12:35:26.508997 2185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:35:26.546938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36041
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:26.661949 2197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:26.662285 2197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:26.662366 2197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:26.666617 2197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:26.666731 2197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:26.666810 2197 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 12:35:26.666854 2197 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:35:26.666923 2197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:35:26.672421 2197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:26.673635 2197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:26.674747 2197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:26.684922 2203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:26.684963 2202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:26.684918 2205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:26.685487 2197 server_base.cc:1034] running on GCE node
I20260619 12:35:26.685987 2197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:26.686707 2197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:26.687896 2197 hybrid_clock.cc:648] HybridClock initialized: now 1781872526687868 us; error 69 us; skew 500 ppm
Jun 19 12:35:26 dist-test-slave-kdrb krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:26.690811 2197 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:35:26.691874 2197 webserver.cc:460] Webserver started at http://127.0.98.1:32975/ using document root <none> and password file <none>
I20260619 12:35:26.692487 2197 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:26.692595 2197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:26.692839 2197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:26.695117 2197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "fb05a9d3c2b94b2d861de43c37911a28"
format_stamp: "Formatted at 2026-06-19 12:35:26 on dist-test-slave-kdrb"
server_key: "678a703db66245804af2489be1e1d2e8"
server_key_iv: "4bd06d6b6532b89261315c8ac00e3e17"
server_key_version: "encryptionkey@0"
I20260619 12:35:26.695779 2197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fb05a9d3c2b94b2d861de43c37911a28"
format_stamp: "Formatted at 2026-06-19 12:35:26 on dist-test-slave-kdrb"
server_key: "678a703db66245804af2489be1e1d2e8"
server_key_iv: "4bd06d6b6532b89261315c8ac00e3e17"
server_key_version: "encryptionkey@0"
I20260619 12:35:26.699774 2197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 12:35:26.702308 2212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:26.703613 2197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:35:26.703743 2197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "fb05a9d3c2b94b2d861de43c37911a28"
format_stamp: "Formatted at 2026-06-19 12:35:26 on dist-test-slave-kdrb"
server_key: "678a703db66245804af2489be1e1d2e8"
server_key_iv: "4bd06d6b6532b89261315c8ac00e3e17"
server_key_version: "encryptionkey@0"
I20260619 12:35:26.703859 2197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:26.739856 2197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:26.743366 2197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:26.743614 2197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:26.745167 2197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:26.745239 2197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:26.745313 2197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:26.745355 2197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:35:26 dist-test-slave-kdrb krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:26.757035 2197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41107
I20260619 12:35:26.757114 2327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41107 every 8 connection(s)
I20260619 12:35:26.758203 2197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:35:26.759603 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:26.746017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49487 (local address 127.0.98.62:36041)
0619 12:35:26.746142 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:26.746146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:26.747259 (+ 1113us) server_negotiation.cc:407] Connection header received
0619 12:35:26.748321 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:26.748325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:26.748389 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:26.748453 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:26.750031 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.750701 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.752031 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.752362 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.752502 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:26.755584 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:26.755592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:26.755596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:26.755632 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:26.757368 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:26.757881 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:26.757885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:26.757887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:26.757985 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:26.758337 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:26.758340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:26.758342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:26.758584 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:26.758693 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:26.759275 (+ 582us) server_negotiation.cc:299] Negotiation successful
0619 12:35:26.759406 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:35:26.760413 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:26.746597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36041 (local address 127.0.98.1:49487)
0619 12:35:26.747099 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:26.747142 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:26.748050 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:26.748810 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:26.748816 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:26.749178 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:26.749832 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:26.749847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.750833 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.750837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:26.751882 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:26.751899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.752552 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.752556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:26.752769 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:26.753462 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:26.753489 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:26.755359 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:26.757484 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:26.757488 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:26.757501 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:26.757756 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:26.758074 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:26.758076 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:26.758078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:26.758206 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:26.758697 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:26.758705 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:26.758997 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 12:35:26.759216 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 12:35:26.759581 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":143,"threads_started":1}
I20260619 12:35:26.761802 2328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36041
I20260619 12:35:26.761967 2328 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:26.762519 2328 heartbeater.cc:511] Master 127.0.98.62:36041 requested a full tablet report, sending...
I20260619 12:35:26.764145 2124 ts_manager.cc:194] Registered new tserver with Master: fb05a9d3c2b94b2d861de43c37911a28 (127.0.98.1:41107)
I20260619 12:35:26.765226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 2197
I20260619 12:35:26.765347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:35:26.765621 392 external_mini_cluster.cc:1428] Setting key 4da05a179c486faa60d862b1cbcbf8c2
I20260619 12:35:26.765621 2124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49487
I20260619 12:35:26.772655 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:26.763854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36041 (local address 127.0.98.1:50301)
0619 12:35:26.764246 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:26.764261 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:26.764357 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:26.764601 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:26.764603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:26.764821 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:26.764937 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:26.764945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.765959 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.765967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:26.766803 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:26.766811 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.767405 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.767412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:26.767570 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:26.768163 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:26.768183 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:26.768687 (+ 504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:26.770937 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:26.770940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:26.770942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:26.771334 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:26.771665 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:26.771668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:26.771669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:26.771747 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:26.772218 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:26.772220 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:26.772298 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 12:35:26.772377 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 12:35:26.772502 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":88,"threads_started":1}
I20260619 12:35:26.772878 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:26.763966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50301 (local address 127.0.98.62:36041)
0619 12:35:26.764090 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:26.764094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:26.764320 (+ 226us) server_negotiation.cc:407] Connection header received
0619 12:35:26.764448 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:26.764450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:26.764489 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:26.764560 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:26.765060 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.765826 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.766964 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.767275 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.767382 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:26.768824 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:26.768830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:26.768834 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:26.768874 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:26.770804 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:26.771448 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:26.771453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:26.771455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:26.771570 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:26.771840 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:26.771846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:26.771851 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:26.772120 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:26.772239 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:26.772477 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 12:35:26.772637 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:35:26.794426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36041
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:26.908504 2337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:26.908785 2337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:26.908834 2337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:26.912985 2337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:26.913049 2337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:26.913090 2337 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 12:35:26.913105 2337 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:35:26.913137 2337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:35:26.918380 2337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:26.919406 2337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:26.920481 2337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:26.930378 2345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:26.930398 2342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:26.930436 2343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:26.930632 2337 server_base.cc:1034] running on GCE node
I20260619 12:35:26.931186 2337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:26.931876 2337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:26.933064 2337 hybrid_clock.cc:648] HybridClock initialized: now 1781872526933050 us; error 40 us; skew 500 ppm
Jun 19 12:35:26 dist-test-slave-kdrb krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:26.935809 2337 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:35:26.936730 2337 webserver.cc:460] Webserver started at http://127.0.98.2:46009/ using document root <none> and password file <none>
I20260619 12:35:26.937317 2337 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:26.937409 2337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:26.937638 2337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:26.939786 2337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "ace8d9dcdacd4dbc871e6a0a058997a5"
format_stamp: "Formatted at 2026-06-19 12:35:26 on dist-test-slave-kdrb"
server_key: "98dfab2c12f8f5a9e7917d209fbbdb02"
server_key_iv: "8b502a00d769353d7c5d842991f297ad"
server_key_version: "encryptionkey@0"
I20260619 12:35:26.940382 2337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ace8d9dcdacd4dbc871e6a0a058997a5"
format_stamp: "Formatted at 2026-06-19 12:35:26 on dist-test-slave-kdrb"
server_key: "98dfab2c12f8f5a9e7917d209fbbdb02"
server_key_iv: "8b502a00d769353d7c5d842991f297ad"
server_key_version: "encryptionkey@0"
I20260619 12:35:26.944294 2337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 12:35:26.946798 2352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:26.948168 2337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 12:35:26.948300 2337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "ace8d9dcdacd4dbc871e6a0a058997a5"
format_stamp: "Formatted at 2026-06-19 12:35:26 on dist-test-slave-kdrb"
server_key: "98dfab2c12f8f5a9e7917d209fbbdb02"
server_key_iv: "8b502a00d769353d7c5d842991f297ad"
server_key_version: "encryptionkey@0"
I20260619 12:35:26.948444 2337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:26.961745 2337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:26.964694 2337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:26.964848 2337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:26.966269 2337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:26.966324 2337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:26.966363 2337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:26.966419 2337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:35:26 dist-test-slave-kdrb krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:26.977741 2337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33813
I20260619 12:35:26.978811 2337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:35:26.979707 2467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33813 every 8 connection(s)
I20260619 12:35:26.980471 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:26.967003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44563 (local address 127.0.98.62:36041)
0619 12:35:26.967157 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:26.967162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:26.968120 (+ 958us) server_negotiation.cc:407] Connection header received
0619 12:35:26.969112 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:26.969116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:26.969178 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:26.969239 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:26.970908 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.971585 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.972837 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.973146 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.973263 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:26.976580 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:26.976585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:26.976588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:26.976619 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:26.978216 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:26.978788 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:26.978791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:26.978792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:26.978893 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:26.979252 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:26.979257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:26.979259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:26.979457 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:26.979593 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:26.980244 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 12:35:26.980350 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 12:35:26.981005 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 2337
I20260619 12:35:26.981115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:35:26.981386 392 external_mini_cluster.cc:1428] Setting key b2f5810638d2df83cdbb570ab591f128
I20260619 12:35:26.981577 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:26.967452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36041 (local address 127.0.98.2:44563)
0619 12:35:26.967951 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:26.968001 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:26.968852 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:26.969809 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:26.969816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:26.970209 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:26.970760 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:26.970777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.971709 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.971714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:26.972708 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:26.972717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.973314 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.973318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:26.973586 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:26.974354 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:26.974379 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:26.976352 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:26.978332 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:26.978338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:26.978353 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:26.978678 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:26.979012 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:26.979014 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:26.979016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:26.979154 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:26.979644 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:26.979652 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:26.979975 (+ 323us) client_negotiation.cc:769] Sending connection context
0619 12:35:26.980205 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 12:35:26.980479 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":124,"threads_started":1}
I20260619 12:35:26.982950 2468 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36041
I20260619 12:35:26.983096 2468 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:26.983672 2468 heartbeater.cc:511] Master 127.0.98.62:36041 requested a full tablet report, sending...
I20260619 12:35:26.984784 2125 ts_manager.cc:194] Registered new tserver with Master: ace8d9dcdacd4dbc871e6a0a058997a5 (127.0.98.2:33813)
I20260619 12:35:26.985433 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44563
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:35:26.992771 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:26.985464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52975 (local address 127.0.98.62:36041)
0619 12:35:26.985591 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:26.985597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:26.985848 (+ 251us) server_negotiation.cc:407] Connection header received
0619 12:35:26.985997 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:26.986000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:26.986045 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:26.986139 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:26.986599 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.987109 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.988001 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.988235 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.988344 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:26.989498 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:26.989501 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:26.989505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:26.989528 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:26.990992 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:26.991415 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:26.991421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:26.991425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:26.991525 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:26.991811 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:26.991816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:26.991817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:26.992066 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:26.992178 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:26.992486 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 12:35:26.992616 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:35:26.992777 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:26.985342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36041 (local address 127.0.98.2:52975)
0619 12:35:26.985755 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:26.985770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:26.985876 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:26.986154 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:26.986158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:26.986378 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:26.986484 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:26.986490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.987233 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.987236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:26.987884 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:26.987892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:26.988347 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:26.988351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:26.988452 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:26.989004 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:26.989016 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:26.989352 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:26.991114 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:26.991117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:26.991119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:26.991320 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:26.991628 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:26.991632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:26.991634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:26.991719 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:26.992167 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:26.992170 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:26.992273 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 12:35:26.992379 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 12:35:26.992561 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:35:27.011521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36041
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:27.127306 2477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:27.127676 2477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:27.127730 2477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:27.131917 2477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:27.131987 2477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:27.132030 2477 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 12:35:27.132045 2477 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:35:27.132077 2477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:35:27.137671 2477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:27.138737 2477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:27.139830 2477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:27.149933 2482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:27.149881 2485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:27.150064 2477 server_base.cc:1034] running on GCE node
W20260619 12:35:27.150254 2483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:27.150715 2477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:27.151538 2477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:27.152705 2477 hybrid_clock.cc:648] HybridClock initialized: now 1781872527152688 us; error 33 us; skew 500 ppm
Jun 19 12:35:27 dist-test-slave-kdrb krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:27.155606 2477 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:35:27.156622 2477 webserver.cc:460] Webserver started at http://127.0.98.3:33927/ using document root <none> and password file <none>
I20260619 12:35:27.157227 2477 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:27.157292 2477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:27.157547 2477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:27.159780 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "ccf458c0235e462a9ac16b672793b8d0"
format_stamp: "Formatted at 2026-06-19 12:35:27 on dist-test-slave-kdrb"
server_key: "96dd00b7d90d563e6e04a2c5d49e85fd"
server_key_iv: "873425abe45b7fe6f99ec4fbdf57d895"
server_key_version: "encryptionkey@0"
I20260619 12:35:27.160403 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ccf458c0235e462a9ac16b672793b8d0"
format_stamp: "Formatted at 2026-06-19 12:35:27 on dist-test-slave-kdrb"
server_key: "96dd00b7d90d563e6e04a2c5d49e85fd"
server_key_iv: "873425abe45b7fe6f99ec4fbdf57d895"
server_key_version: "encryptionkey@0"
I20260619 12:35:27.164479 2477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:35:27.167241 2492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:27.168952 2477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:35:27.169087 2477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "ccf458c0235e462a9ac16b672793b8d0"
format_stamp: "Formatted at 2026-06-19 12:35:27 on dist-test-slave-kdrb"
server_key: "96dd00b7d90d563e6e04a2c5d49e85fd"
server_key_iv: "873425abe45b7fe6f99ec4fbdf57d895"
server_key_version: "encryptionkey@0"
I20260619 12:35:27.169219 2477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:27.184224 2477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:27.187461 2477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:27.187706 2477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:27.189147 2477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:27.189199 2477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:27.189241 2477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:27.189302 2477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:35:27 dist-test-slave-kdrb krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:27.200212 2477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46595
I20260619 12:35:27.200282 2607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46595 every 8 connection(s)
I20260619 12:35:27.201212 2477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:35:27.204699 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.190069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58285 (local address 127.0.98.62:36041)
0619 12:35:27.190194 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.190198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.191337 (+ 1139us) server_negotiation.cc:407] Connection header received
0619 12:35:27.192629 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.192633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.192695 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.192817 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:27.194714 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.195613 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.197017 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.197337 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.197460 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.200644 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:27.200647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:27.200650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:27.200676 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:27.202480 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:27.203080 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:27.203083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:27.203085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:27.203175 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:27.203576 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:27.203579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:27.203582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:27.203778 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:27.203886 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.204391 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.204529 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 12:35:27.205530 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.190679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36041 (local address 127.0.98.3:58285)
0619 12:35:27.191181 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:27.191225 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:27.192337 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:27.193085 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:27.193099 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:27.193637 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:27.194514 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.194540 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.195803 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.195807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.196868 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.196879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.197511 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.197516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.197704 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.198378 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:27.198397 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:27.200464 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:27.202600 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:27.202607 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:27.202627 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:27.202961 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:27.203270 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:27.203273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:27.203275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:27.203427 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:27.203884 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:27.203891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:27.204148 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 12:35:27.204335 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 12:35:27.204670 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":152,"threads_started":1}
I20260619 12:35:27.206949 2608 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36041
I20260619 12:35:27.207103 2608 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:27.207733 2608 heartbeater.cc:511] Master 127.0.98.62:36041 requested a full tablet report, sending...
I20260619 12:35:27.208415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 2477
I20260619 12:35:27.208561 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:35:27.208886 392 external_mini_cluster.cc:1428] Setting key bcf72a9df3277c14442e88effeb4afd7
I20260619 12:35:27.208952 2125 ts_manager.cc:194] Registered new tserver with Master: ccf458c0235e462a9ac16b672793b8d0 (127.0.98.3:46595)
I20260619 12:35:27.209757 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58285
I20260619 12:35:27.211226 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:35:27.217708 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.209392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36041 (local address 127.0.98.3:42077)
0619 12:35:27.209656 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:27.209671 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:27.209773 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:27.210195 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:27.210198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:27.210435 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:27.210589 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.210597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.211698 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.211702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.212560 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.212572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.213250 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.213252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.213317 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.213855 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:27.213876 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:27.214263 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:27.216119 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:27.216123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:27.216125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:27.216403 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:27.216721 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:27.216724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:27.216725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:27.216801 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:27.217279 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:27.217281 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:27.217357 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 12:35:27.217436 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 12:35:27.217564 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":3,"thread_start_us":98,"threads_started":1}
I20260619 12:35:27.217914 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.209446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42077 (local address 127.0.98.62:36041)
0619 12:35:27.209663 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.209666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.209886 (+ 220us) server_negotiation.cc:407] Connection header received
0619 12:35:27.209928 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.209932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.209978 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.210063 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:27.210718 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.211447 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.212688 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.213021 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.213104 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.214405 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:27.214408 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:27.214409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:27.214432 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:27.216011 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:27.216529 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:27.216533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:27.216536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:27.216623 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:27.216894 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:27.216897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:27.216899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:27.217181 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:27.217292 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.217523 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.217664 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
Jun 19 12:35:27 dist-test-slave-kdrb krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 12:35:27 dist-test-slave-kdrb krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:27.238829 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.230100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.0.98.62:36041)
0619 12:35:27.230234 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.230237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.230250 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:35:27.230291 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.230294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.230332 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.230430 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:27.231178 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.231771 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.232641 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.232860 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.232974 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.235459 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:27.235522 (+ 63us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:27.235526 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:27.235563 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:27.237057 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:27.237512 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:27.237518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:27.237520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:27.237604 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:27.237893 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:27.237896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:27.237898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:27.238122 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:27.238268 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.238509 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.238652 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:35:27.242010 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:35:27.245182 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:27.259635 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.253976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41107 (local address 127.0.0.1:35788)
0619 12:35:27.254461 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:27.254517 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:27.254650 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:27.255534 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:27.255540 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:27.255570 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:27.255716 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.255725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.256978 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.256982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.257941 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.257960 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.259025 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.259029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.259165 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.259210 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 12:35:27.259326 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 12:35:27.259446 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":109,"threads_started":1}
I20260619 12:35:27.259653 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.254157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35788 (local address 127.0.98.1:41107)
0619 12:35:27.254532 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.254548 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.254579 (+ 31us) server_negotiation.cc:407] Connection header received
0619 12:35:27.254827 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.254834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.255070 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.255242 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:27.255849 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.256863 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.258076 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.258902 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.259012 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.259204 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.259372 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.259499 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":83,"threads_started":1}
I20260619 12:35:27.260589 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.254231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33813 (local address 127.0.0.1:39922)
0619 12:35:27.255025 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:27.255038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:27.255132 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:27.256033 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:27.256036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:27.256053 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:27.256147 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.256155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.257789 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.257793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.258899 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.258911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.260191 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.260194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.260281 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.260303 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:35:27.260350 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 12:35:27.260430 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":32,"thread_start_us":128,"threads_started":1}
I20260619 12:35:27.260721 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.254723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46595 (local address 127.0.0.1:59886)
0619 12:35:27.255543 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:27.255557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:27.255644 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:27.256319 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:27.256322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:27.256340 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:27.256437 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.256443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.257820 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.257823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.259014 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.259024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.260237 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.260241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.260344 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.260366 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:35:27.260465 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 12:35:27.260544 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":23,"thread_start_us":93,"threads_started":1}
I20260619 12:35:27.260730 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.254904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.0.98.3:46595)
0619 12:35:27.255878 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.255901 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.255925 (+ 24us) server_negotiation.cc:407] Connection header received
0619 12:35:27.255999 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.256007 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.256177 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.256312 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:27.256559 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.257666 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.259130 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.260009 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.260142 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.260314 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.260499 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.260590 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":118,"threads_started":1}
I20260619 12:35:27.261438 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.255014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.0.98.2:33813)
0619 12:35:27.255561 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.255577 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.255602 (+ 25us) server_negotiation.cc:407] Connection header received
0619 12:35:27.255677 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.255685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.255871 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.256295 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:27.256433 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.257663 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.259170 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.260046 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.260953 (+ 907us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.261149 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.261206 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.261292 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":121,"threads_started":1}
I20260619 12:35:27.263214 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 00bb11027b4d41bd9511bf77d581130e (DEFAULT_TABLE table=test-table [id=859d279b14144aada0c06b6717c5be51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:27.263204 2262 tablet_service.cc:1467] Processing CreateTablet for tablet 00bb11027b4d41bd9511bf77d581130e (DEFAULT_TABLE table=test-table [id=859d279b14144aada0c06b6717c5be51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:27.264032 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 00bb11027b4d41bd9511bf77d581130e (DEFAULT_TABLE table=test-table [id=859d279b14144aada0c06b6717c5be51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:27.264384 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00bb11027b4d41bd9511bf77d581130e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:27.264379 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00bb11027b4d41bd9511bf77d581130e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:27.264873 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00bb11027b4d41bd9511bf77d581130e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:27.272184 2631 tablet_bootstrap.cc:492] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5: Bootstrap starting.
I20260619 12:35:27.272629 2632 tablet_bootstrap.cc:492] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28: Bootstrap starting.
I20260619 12:35:27.273110 2633 tablet_bootstrap.cc:492] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0: Bootstrap starting.
I20260619 12:35:27.275364 2633 tablet_bootstrap.cc:654] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:27.275364 2632 tablet_bootstrap.cc:654] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:27.276341 2633 log.cc:824] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:27.277261 2631 tablet_bootstrap.cc:654] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:27.278051 2631 log.cc:824] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:27.278569 2633 tablet_bootstrap.cc:492] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0: No bootstrap required, opened a new log
I20260619 12:35:27.278797 2633 ts_tablet_manager.cc:1397] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:35:27.279726 2632 log.cc:824] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:27.280098 2631 tablet_bootstrap.cc:492] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5: No bootstrap required, opened a new log
I20260619 12:35:27.280296 2631 ts_tablet_manager.cc:1397] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20260619 12:35:27.281703 2632 tablet_bootstrap.cc:492] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28: No bootstrap required, opened a new log
I20260619 12:35:27.281893 2632 ts_tablet_manager.cc:1397] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260619 12:35:27.283377 2633 raft_consensus.cc:348] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.283694 2633 raft_consensus.cc:374] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:27.283747 2633 raft_consensus.cc:729] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf458c0235e462a9ac16b672793b8d0, State: Initialized, Role: FOLLOWER
I20260619 12:35:27.283808 2631 raft_consensus.cc:348] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.284075 2631 raft_consensus.cc:374] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:27.284124 2631 raft_consensus.cc:729] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace8d9dcdacd4dbc871e6a0a058997a5, State: Initialized, Role: FOLLOWER
I20260619 12:35:27.284268 2633 consensus_queue.cc:260] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.284520 2631 consensus_queue.cc:260] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.285362 2631 ts_tablet_manager.cc:1428] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260619 12:35:27.285368 2633 ts_tablet_manager.cc:1428] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 12:35:27.285727 2608 heartbeater.cc:503] Master 127.0.98.62:36041 was elected leader, sending a full tablet report...
I20260619 12:35:27.285727 2468 heartbeater.cc:503] Master 127.0.98.62:36041 was elected leader, sending a full tablet report...
I20260619 12:35:27.285588 2632 raft_consensus.cc:348] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.285821 2632 raft_consensus.cc:374] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:27.285873 2632 raft_consensus.cc:729] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb05a9d3c2b94b2d861de43c37911a28, State: Initialized, Role: FOLLOWER
I20260619 12:35:27.286290 2632 consensus_queue.cc:260] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.286881 2328 heartbeater.cc:503] Master 127.0.98.62:36041 was elected leader, sending a full tablet report...
I20260619 12:35:27.288079 2632 ts_tablet_manager.cc:1428] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260619 12:35:27.452283 2609 tablet.cc:2369] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:35:27.480078 2469 tablet.cc:2369] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:35:27.491412 2638 raft_consensus.cc:482] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:27.491676 2638 raft_consensus.cc:504] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.492864 2638 leader_election.cc:288] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb05a9d3c2b94b2d861de43c37911a28 (127.0.98.1:41107), ccf458c0235e462a9ac16b672793b8d0 (127.0.98.3:46595)
I20260619 12:35:27.496912 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.493204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46595 (local address 127.0.98.2:45109)
0619 12:35:27.493577 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:27.493589 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:27.493697 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:27.493986 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:27.493988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:27.494002 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:27.494084 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.494088 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.494855 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.494858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.495799 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.495810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.496456 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.496460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.496553 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.496579 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 12:35:27.496654 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 12:35:27.496726 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":63,"thread_start_us":63,"threads_started":1}
I20260619 12:35:27.496925 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.493491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45109 (local address 127.0.98.3:46595)
0619 12:35:27.493643 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.493650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.493668 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:35:27.493799 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.493802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.493875 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.493955 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:27.494214 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.494723 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.495948 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.496342 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.496424 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.496477 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.496682 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.496755 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 12:35:27.497123 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.493205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48053 (local address 127.0.98.1:41107)
0619 12:35:27.493376 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.493381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.493466 (+ 85us) server_negotiation.cc:407] Connection header received
0619 12:35:27.493717 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.493723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.493787 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.493858 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:27.494169 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.494705 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.495910 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.496562 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.496674 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.496731 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.496917 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.497003 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:35:27.497313 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.493056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41107 (local address 127.0.98.2:48053)
0619 12:35:27.493366 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:27.493382 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:27.493582 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:27.493891 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:27.493895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:27.493917 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:27.494063 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.494069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.494817 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.494820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.495795 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.495804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.496694 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.496697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.496787 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.496809 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:35:27.496887 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 12:35:27.496960 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":5,"thread_start_us":95,"threads_started":1}
I20260619 12:35:27.497726 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bb11027b4d41bd9511bf77d581130e" candidate_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf458c0235e462a9ac16b672793b8d0" is_pre_election: true
I20260619 12:35:27.497897 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bb11027b4d41bd9511bf77d581130e" candidate_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb05a9d3c2b94b2d861de43c37911a28" is_pre_election: true
I20260619 12:35:27.498064 2562 raft_consensus.cc:2454] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ace8d9dcdacd4dbc871e6a0a058997a5 in term 0.
I20260619 12:35:27.498207 2282 raft_consensus.cc:2454] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ace8d9dcdacd4dbc871e6a0a058997a5 in term 0.
I20260619 12:35:27.498555 2353 leader_election.cc:302] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ace8d9dcdacd4dbc871e6a0a058997a5, ccf458c0235e462a9ac16b672793b8d0; no voters:
I20260619 12:35:27.498849 2638 raft_consensus.cc:2781] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:27.498934 2638 raft_consensus.cc:482] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:27.498983 2638 raft_consensus.cc:3037] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:27.500051 2638 raft_consensus.cc:504] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.500451 2638 leader_election.cc:288] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [CANDIDATE]: Term 1 election: Requested vote from peers fb05a9d3c2b94b2d861de43c37911a28 (127.0.98.1:41107), ccf458c0235e462a9ac16b672793b8d0 (127.0.98.3:46595)
I20260619 12:35:27.500802 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bb11027b4d41bd9511bf77d581130e" candidate_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf458c0235e462a9ac16b672793b8d0"
I20260619 12:35:27.500802 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bb11027b4d41bd9511bf77d581130e" candidate_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb05a9d3c2b94b2d861de43c37911a28"
I20260619 12:35:27.500932 2562 raft_consensus.cc:3037] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:27.500932 2282 raft_consensus.cc:3037] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:27.502287 2282 raft_consensus.cc:2454] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ace8d9dcdacd4dbc871e6a0a058997a5 in term 1.
I20260619 12:35:27.502627 2356 leader_election.cc:302] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ace8d9dcdacd4dbc871e6a0a058997a5, fb05a9d3c2b94b2d861de43c37911a28; no voters:
I20260619 12:35:27.502713 2562 raft_consensus.cc:2454] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ace8d9dcdacd4dbc871e6a0a058997a5 in term 1.
I20260619 12:35:27.502851 2638 raft_consensus.cc:2781] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:27.503034 2638 raft_consensus.cc:686] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 LEADER]: Becoming Leader. State: Replica: ace8d9dcdacd4dbc871e6a0a058997a5, State: Running, Role: LEADER
I20260619 12:35:27.503383 2638 consensus_queue.cc:237] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.506845 2125 catalog_manager.cc:5495] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ace8d9dcdacd4dbc871e6a0a058997a5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } health_report { overall_health: UNKNOWN } } }
W20260619 12:35:27.510442 2329 tablet.cc:2369] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 12:35:27 dist-test-slave-kdrb krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:27.574047 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.565315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36041 (local address 127.0.0.1:55068)
0619 12:35:27.565628 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:27.565643 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:27.565788 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:27.566030 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:27.566033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:27.566448 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:27.566553 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.566559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.567290 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.567292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.567905 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.567915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.568317 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.568319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.568384 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.569054 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:27.569068 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:27.570550 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:27.572344 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:27.572351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:27.572358 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:27.572661 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:27.572929 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:27.572931 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:27.572932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:27.573013 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:27.573474 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:27.573479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:27.573587 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 12:35:27.573708 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 12:35:27.573863 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20260619 12:35:27.574054 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.565427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55068 (local address 127.0.98.62:36041)
0619 12:35:27.565608 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.565613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.565723 (+ 110us) server_negotiation.cc:407] Connection header received
0619 12:35:27.565872 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.565874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.565915 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.565986 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:27.566658 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.567166 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.568010 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.568213 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.568290 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.570713 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:27.570716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:27.570718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:27.570739 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:27.572232 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:27.572753 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:27.572756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:27.572757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:27.572835 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:27.573120 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:27.573153 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:27.573160 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:27.573375 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:27.573466 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.573755 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.573925 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":97}
I20260619 12:35:27.578074 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55068:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 12:35:27.578845 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:27.584010 2262 tablet_service.cc:1467] Processing CreateTablet for tablet 9d37b04a6ac54b3abd9afea225c73c64 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=916810aa6c874ed3b2e34f59475c9dfc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:35:27.583993 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 9d37b04a6ac54b3abd9afea225c73c64 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=916810aa6c874ed3b2e34f59475c9dfc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:35:27.583982 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 9d37b04a6ac54b3abd9afea225c73c64 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=916810aa6c874ed3b2e34f59475c9dfc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:35:27.584781 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d37b04a6ac54b3abd9afea225c73c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:27.584780 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d37b04a6ac54b3abd9afea225c73c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:27.584780 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d37b04a6ac54b3abd9afea225c73c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:27.587452 2632 tablet_bootstrap.cc:492] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28: Bootstrap starting.
I20260619 12:35:27.587673 2633 tablet_bootstrap.cc:492] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0: Bootstrap starting.
I20260619 12:35:27.587673 2631 tablet_bootstrap.cc:492] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5: Bootstrap starting.
I20260619 12:35:27.588552 2632 tablet_bootstrap.cc:654] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:27.588721 2633 tablet_bootstrap.cc:654] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:27.589049 2631 tablet_bootstrap.cc:654] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:27.589952 2632 tablet_bootstrap.cc:492] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28: No bootstrap required, opened a new log
I20260619 12:35:27.590056 2632 ts_tablet_manager.cc:1397] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 12:35:27.590250 2633 tablet_bootstrap.cc:492] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0: No bootstrap required, opened a new log
I20260619 12:35:27.590332 2633 ts_tablet_manager.cc:1397] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:35:27.590585 2632 raft_consensus.cc:348] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.590719 2632 raft_consensus.cc:374] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:27.590760 2632 raft_consensus.cc:729] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb05a9d3c2b94b2d861de43c37911a28, State: Initialized, Role: FOLLOWER
I20260619 12:35:27.590803 2633 raft_consensus.cc:348] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.590924 2633 raft_consensus.cc:374] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:27.590958 2633 raft_consensus.cc:729] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf458c0235e462a9ac16b672793b8d0, State: Initialized, Role: FOLLOWER
I20260619 12:35:27.590927 2632 consensus_queue.cc:260] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.591087 2633 consensus_queue.cc:260] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.591212 2632 ts_tablet_manager.cc:1428] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:35:27.591207 2639 tablet_replica.cc:439] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } } }
I20260619 12:35:27.591342 2639 tablet_replica.cc:442] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:27.591337 2633 ts_tablet_manager.cc:1428] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:35:27.591599 2637 tablet_replica.cc:439] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } } }
I20260619 12:35:27.591719 2637 tablet_replica.cc:442] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:27.592342 2631 tablet_bootstrap.cc:492] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5: No bootstrap required, opened a new log
I20260619 12:35:27.592439 2631 ts_tablet_manager.cc:1397] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260619 12:35:27.592847 2631 raft_consensus.cc:348] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.592964 2631 raft_consensus.cc:374] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:27.593005 2631 raft_consensus.cc:729] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace8d9dcdacd4dbc871e6a0a058997a5, State: Initialized, Role: FOLLOWER
I20260619 12:35:27.593133 2631 consensus_queue.cc:260] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.593479 2631 ts_tablet_manager.cc:1428] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260619 12:35:27.593423 2638 tablet_replica.cc:439] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } } }
I20260619 12:35:27.593557 2638 tablet_replica.cc:442] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:27.616623 2638 raft_consensus.cc:482] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:27.616746 2638 raft_consensus.cc:504] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.617192 2638 leader_election.cc:288] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb05a9d3c2b94b2d861de43c37911a28 (127.0.98.1:41107), ccf458c0235e462a9ac16b672793b8d0 (127.0.98.3:46595)
I20260619 12:35:27.617614 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d37b04a6ac54b3abd9afea225c73c64" candidate_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb05a9d3c2b94b2d861de43c37911a28" is_pre_election: true
I20260619 12:35:27.617745 2282 raft_consensus.cc:2454] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ace8d9dcdacd4dbc871e6a0a058997a5 in term 0.
I20260619 12:35:27.617678 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d37b04a6ac54b3abd9afea225c73c64" candidate_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf458c0235e462a9ac16b672793b8d0" is_pre_election: true
I20260619 12:35:27.617887 2562 raft_consensus.cc:2454] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ace8d9dcdacd4dbc871e6a0a058997a5 in term 0.
I20260619 12:35:27.618063 2356 leader_election.cc:302] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ace8d9dcdacd4dbc871e6a0a058997a5, fb05a9d3c2b94b2d861de43c37911a28; no voters:
I20260619 12:35:27.618260 2638 raft_consensus.cc:2781] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:27.618336 2638 raft_consensus.cc:482] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:27.618405 2638 raft_consensus.cc:3037] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:27.619527 2638 raft_consensus.cc:504] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.619987 2638 leader_election.cc:288] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [CANDIDATE]: Term 1 election: Requested vote from peers fb05a9d3c2b94b2d861de43c37911a28 (127.0.98.1:41107), ccf458c0235e462a9ac16b672793b8d0 (127.0.98.3:46595)
I20260619 12:35:27.620308 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d37b04a6ac54b3abd9afea225c73c64" candidate_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb05a9d3c2b94b2d861de43c37911a28"
I20260619 12:35:27.620447 2282 raft_consensus.cc:3037] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:27.621178 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d37b04a6ac54b3abd9afea225c73c64" candidate_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf458c0235e462a9ac16b672793b8d0"
I20260619 12:35:27.621289 2562 raft_consensus.cc:3037] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:27.621587 2282 raft_consensus.cc:2454] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ace8d9dcdacd4dbc871e6a0a058997a5 in term 1.
I20260619 12:35:27.621861 2356 leader_election.cc:302] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ace8d9dcdacd4dbc871e6a0a058997a5, fb05a9d3c2b94b2d861de43c37911a28; no voters:
I20260619 12:35:27.622045 2638 raft_consensus.cc:2781] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:27.622138 2638 raft_consensus.cc:686] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 LEADER]: Becoming Leader. State: Replica: ace8d9dcdacd4dbc871e6a0a058997a5, State: Running, Role: LEADER
I20260619 12:35:27.622285 2638 consensus_queue.cc:237] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.622435 2562 raft_consensus.cc:2454] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ace8d9dcdacd4dbc871e6a0a058997a5 in term 1.
I20260619 12:35:27.623044 2642 tablet_replica.cc:439] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5: TxnStatusTablet state changed. Reason: New leader ace8d9dcdacd4dbc871e6a0a058997a5. Latest consensus state: current_term: 1 leader_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } } }
I20260619 12:35:27.623148 2642 tablet_replica.cc:442] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:35:27.623447 2654 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 12:35:27.623725 2654 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 12:35:27.624114 2123 catalog_manager.cc:5495] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ace8d9dcdacd4dbc871e6a0a058997a5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } health_report { overall_health: UNKNOWN } } }
I20260619 12:35:27.644797 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.641114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39926 (local address 127.0.98.2:33813)
0619 12:35:27.641232 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.641238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.641256 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:35:27.641492 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.641495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.641549 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.641646 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:27.641893 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.642413 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.643276 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.643560 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.643668 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.643973 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:27.644382 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:27.644523 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.644598 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.644655 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:35:27.644797 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.641006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33813 (local address 127.0.0.1:39926)
0619 12:35:27.641195 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:27.641211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:27.641328 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:27.641670 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:27.641673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:27.641682 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:35:27.641791 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.641796 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.642534 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.642539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.643149 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.643157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.643668 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.643672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.643769 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.643812 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:35:27.644497 (+ 685us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:35:27.644512 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 12:35:27.644579 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 12:35:27.644669 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 12:35:27.650324 2562 raft_consensus.cc:1261] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Refusing update from remote peer ace8d9dcdacd4dbc871e6a0a058997a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:27.650324 2282 raft_consensus.cc:1261] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28 [term 1 FOLLOWER]: Refusing update from remote peer ace8d9dcdacd4dbc871e6a0a058997a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:27.651017 2638 consensus_queue.cc:1035] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:27.651206 2642 consensus_queue.cc:1035] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:27.655336 2637 tablet_replica.cc:439] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0: TxnStatusTablet state changed. Reason: New leader ace8d9dcdacd4dbc871e6a0a058997a5. Latest consensus state: current_term: 1 leader_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } } }
I20260619 12:35:27.655490 2637 tablet_replica.cc:442] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:27.655839 2639 tablet_replica.cc:439] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28: TxnStatusTablet state changed. Reason: New leader ace8d9dcdacd4dbc871e6a0a058997a5. Latest consensus state: current_term: 1 leader_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } } }
I20260619 12:35:27.656351 2639 tablet_replica.cc:442] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:27.657359 2638 tablet_replica.cc:439] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } } }
I20260619 12:35:27.657490 2638 tablet_replica.cc:442] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:35:27.659134 2637 tablet_replica.cc:439] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } } }
I20260619 12:35:27.659256 2637 tablet_replica.cc:442] T 9d37b04a6ac54b3abd9afea225c73c64 P ccf458c0235e462a9ac16b672793b8d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:27.660490 2657 mvcc.cc:205] Tried to move back new op lower bound from 7298549873248530432 to 7298549873142431744. Current Snapshot: MvccSnapshot[applied={T|T < 7298549873248530432 or (T in {7298549873248530432})}]
I20260619 12:35:27.663931 2638 tablet_replica.cc:439] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } } }
I20260619 12:35:27.664079 2638 tablet_replica.cc:442] T 9d37b04a6ac54b3abd9afea225c73c64 P ace8d9dcdacd4dbc871e6a0a058997a5: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:35:27.665324 2639 tablet_replica.cc:439] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } } }
I20260619 12:35:27.665453 2639 tablet_replica.cc:442] T 9d37b04a6ac54b3abd9afea225c73c64 P fb05a9d3c2b94b2d861de43c37911a28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:27.665498 2658 mvcc.cc:205] Tried to move back new op lower bound from 7298549873248530432 to 7298549873142431744. Current Snapshot: MvccSnapshot[applied={T|T < 7298549873248530432}]
I20260619 12:35:27.668234 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.663867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.0.98.2:33813)
0619 12:35:27.663983 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.663987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.664327 (+ 340us) server_negotiation.cc:407] Connection header received
0619 12:35:27.664377 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.664380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.664432 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.664497 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:27.664859 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.665644 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.666991 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.667351 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.667464 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.667785 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:27.667870 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:27.667943 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.668044 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.668098 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 12:35:27.680202 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.676528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50641 (local address 127.0.98.2:33813)
0619 12:35:27.676656 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.676661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.676835 (+ 174us) server_negotiation.cc:407] Connection header received
0619 12:35:27.676932 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.676935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.676983 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.677066 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:27.677300 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.678060 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.678920 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.679273 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.679380 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.679738 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:27.679824 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:27.679914 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.679988 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.680043 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":45}
I20260619 12:35:27.680204 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.676438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33813 (local address 127.0.98.2:50641)
0619 12:35:27.676746 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:27.676768 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:27.676862 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:27.677084 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:27.677086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:27.677091 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:35:27.677193 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.677197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.678178 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.678180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.678813 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.678820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.679386 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.679389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.679545 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.679589 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:35:27.679916 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:35:27.679921 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:35:27.679977 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 12:35:27.680046 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":4,"thread_start_us":107,"threads_started":1}
I20260619 12:35:27.687362 2282 raft_consensus.cc:1261] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28 [term 1 FOLLOWER]: Refusing update from remote peer ace8d9dcdacd4dbc871e6a0a058997a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:27.687502 2562 raft_consensus.cc:1261] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Refusing update from remote peer ace8d9dcdacd4dbc871e6a0a058997a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:27.687808 2638 consensus_queue.cc:1035] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:27.687973 2642 consensus_queue.cc:1035] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:27.692747 2659 mvcc.cc:205] Tried to move back new op lower bound from 7298549873404223488 to 7298549872656572416. Current Snapshot: MvccSnapshot[applied={T|T < 7298549873404223488}]
I20260619 12:35:27.728904 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:27.729108 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:27.731644 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc: Sending DeleteTablet for 3 replicas of tablet 00bb11027b4d41bd9511bf77d581130e
I20260619 12:35:27.732502 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet 00bb11027b4d41bd9511bf77d581130e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39922
I20260619 12:35:27.732594 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 00bb11027b4d41bd9511bf77d581130e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59886
I20260619 12:35:27.732610 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet 00bb11027b4d41bd9511bf77d581130e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35788
I20260619 12:35:27.733031 2674 tablet_replica.cc:330] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5: stopping tablet replica
I20260619 12:35:27.733328 2674 raft_consensus.cc:2229] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:35:27.733621 2673 tablet_replica.cc:330] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0: stopping tablet replica
I20260619 12:35:27.733747 2674 raft_consensus.cc:2258] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:27.733922 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:35:27.734369 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:27.735085 2674 ts_tablet_manager.cc:1905] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:27.735245 2673 raft_consensus.cc:2229] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:27.735522 2673 raft_consensus.cc:2258] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:27.736918 2673 ts_tablet_manager.cc:1905] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:27.740227 2401 tablet_service.cc:1467] Processing CreateTablet for tablet bfaf38b7edcc46cab4e0dbc832186c18 (DEFAULT_TABLE table=test-table [id=0d938ca41845425a9442a8eb2b1176a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:27.740226 2542 tablet_service.cc:1467] Processing CreateTablet for tablet bfaf38b7edcc46cab4e0dbc832186c18 (DEFAULT_TABLE table=test-table [id=0d938ca41845425a9442a8eb2b1176a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:27.740523 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfaf38b7edcc46cab4e0dbc832186c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:27.740535 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfaf38b7edcc46cab4e0dbc832186c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:27.742517 2631 tablet_bootstrap.cc:492] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5: Bootstrap starting.
I20260619 12:35:27.742957 2262 tablet_service.cc:1467] Processing CreateTablet for tablet bfaf38b7edcc46cab4e0dbc832186c18 (DEFAULT_TABLE table=test-table [id=0d938ca41845425a9442a8eb2b1176a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:27.743223 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfaf38b7edcc46cab4e0dbc832186c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:27.743448 2631 tablet_bootstrap.cc:654] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:27.743610 2674 ts_tablet_manager.cc:1918] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:35:27.743693 2674 log.cc:1196] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/00bb11027b4d41bd9511bf77d581130e
I20260619 12:35:27.744002 2674 ts_tablet_manager.cc:1939] T 00bb11027b4d41bd9511bf77d581130e P ace8d9dcdacd4dbc871e6a0a058997a5: Deleting consensus metadata
I20260619 12:35:27.744328 2675 tablet_replica.cc:330] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28: stopping tablet replica
I20260619 12:35:27.744685 2675 raft_consensus.cc:2229] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:27.744907 2675 raft_consensus.cc:2258] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:27.745687 2632 tablet_bootstrap.cc:492] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28: Bootstrap starting.
I20260619 12:35:27.745779 2633 tablet_bootstrap.cc:492] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0: Bootstrap starting.
I20260619 12:35:27.746233 2109 catalog_manager.cc:4841] TS ace8d9dcdacd4dbc871e6a0a058997a5 (127.0.98.2:33813): tablet 00bb11027b4d41bd9511bf77d581130e (table test-table [id=859d279b14144aada0c06b6717c5be51]) successfully deleted
I20260619 12:35:27.746693 2632 tablet_bootstrap.cc:654] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:27.746731 2633 tablet_bootstrap.cc:654] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:27.747900 2675 ts_tablet_manager.cc:1905] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:27.748225 2673 ts_tablet_manager.cc:1918] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:35:27.748409 2673 log.cc:1196] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/00bb11027b4d41bd9511bf77d581130e
I20260619 12:35:27.748752 2673 ts_tablet_manager.cc:1939] T 00bb11027b4d41bd9511bf77d581130e P ccf458c0235e462a9ac16b672793b8d0: Deleting consensus metadata
I20260619 12:35:27.750248 2109 catalog_manager.cc:4841] TS ccf458c0235e462a9ac16b672793b8d0 (127.0.98.3:46595): tablet 00bb11027b4d41bd9511bf77d581130e (table test-table [id=859d279b14144aada0c06b6717c5be51]) successfully deleted
I20260619 12:35:27.750727 2675 ts_tablet_manager.cc:1918] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:35:27.750842 2675 log.cc:1196] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/wal/wals/00bb11027b4d41bd9511bf77d581130e
I20260619 12:35:27.750856 2633 tablet_bootstrap.cc:492] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0: No bootstrap required, opened a new log
I20260619 12:35:27.750950 2633 ts_tablet_manager.cc:1397] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 12:35:27.751163 2675 ts_tablet_manager.cc:1939] T 00bb11027b4d41bd9511bf77d581130e P fb05a9d3c2b94b2d861de43c37911a28: Deleting consensus metadata
I20260619 12:35:27.751430 2633 raft_consensus.cc:348] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.751605 2633 raft_consensus.cc:374] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:27.751653 2633 raft_consensus.cc:729] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf458c0235e462a9ac16b672793b8d0, State: Initialized, Role: FOLLOWER
I20260619 12:35:27.751791 2633 consensus_queue.cc:260] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.752082 2633 ts_tablet_manager.cc:1428] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 12:35:27.752246 2631 tablet_bootstrap.cc:492] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5: No bootstrap required, opened a new log
I20260619 12:35:27.752347 2631 ts_tablet_manager.cc:1397] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260619 12:35:27.752730 2112 catalog_manager.cc:4841] TS fb05a9d3c2b94b2d861de43c37911a28 (127.0.98.1:41107): tablet 00bb11027b4d41bd9511bf77d581130e (table test-table [id=859d279b14144aada0c06b6717c5be51]) successfully deleted
I20260619 12:35:27.752830 2631 raft_consensus.cc:348] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.752954 2631 raft_consensus.cc:374] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:27.752998 2631 raft_consensus.cc:729] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace8d9dcdacd4dbc871e6a0a058997a5, State: Initialized, Role: FOLLOWER
I20260619 12:35:27.753127 2631 consensus_queue.cc:260] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.753254 2632 tablet_bootstrap.cc:492] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28: No bootstrap required, opened a new log
I20260619 12:35:27.753373 2632 ts_tablet_manager.cc:1397] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20260619 12:35:27.753441 2631 ts_tablet_manager.cc:1428] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:35:27.753839 2632 raft_consensus.cc:348] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.753948 2632 raft_consensus.cc:374] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:27.753988 2632 raft_consensus.cc:729] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb05a9d3c2b94b2d861de43c37911a28, State: Initialized, Role: FOLLOWER
I20260619 12:35:27.754098 2632 consensus_queue.cc:260] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.754362 2632 ts_tablet_manager.cc:1428] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:35:27.910427 2639 raft_consensus.cc:482] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:27.910612 2639 raft_consensus.cc:504] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.912045 2639 leader_election.cc:288] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ace8d9dcdacd4dbc871e6a0a058997a5 (127.0.98.2:33813), ccf458c0235e462a9ac16b672793b8d0 (127.0.98.3:46595)
I20260619 12:35:27.916910 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.912484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46643 (local address 127.0.98.2:33813)
0619 12:35:27.912663 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.912668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.912790 (+ 122us) server_negotiation.cc:407] Connection header received
0619 12:35:27.913043 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.913046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.913102 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.913169 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:27.913568 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.914153 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.915658 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.916279 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.916414 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.916487 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.916697 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.916779 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 12:35:27.916920 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.912341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33813 (local address 127.0.98.1:46643)
0619 12:35:27.912706 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:27.912722 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:27.912883 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:27.913321 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:27.913323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:27.913345 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:27.913422 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.913426 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.914269 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.914271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.915518 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.915529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.916425 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.916428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.916527 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.916583 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 12:35:27.916658 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 12:35:27.916748 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":26,"thread_start_us":83,"threads_started":1}
I20260619 12:35:27.916977 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.912871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53205 (local address 127.0.98.3:46595)
0619 12:35:27.912993 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.912997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.913226 (+ 229us) server_negotiation.cc:407] Connection header received
0619 12:35:27.913321 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.913324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.913380 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.913485 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:27.913768 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.914499 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.915904 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.916302 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.916424 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.916484 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.916772 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.916849 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:35:27.917358 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.912748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46595 (local address 127.0.98.1:53205)
0619 12:35:27.913121 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:27.913137 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:27.913224 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:27.913527 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:27.913532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:27.913553 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:27.913640 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.913645 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.914651 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.914654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.915785 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.915792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.916535 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.916538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.916639 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.916665 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 12:35:27.917105 (+ 440us) client_negotiation.cc:240] Negotiation successful
0619 12:35:27.917180 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":35,"thread_start_us":100,"threads_started":1}
I20260619 12:35:27.917469 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfaf38b7edcc46cab4e0dbc832186c18" candidate_uuid: "fb05a9d3c2b94b2d861de43c37911a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" is_pre_election: true
I20260619 12:35:27.917757 2422 raft_consensus.cc:2454] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb05a9d3c2b94b2d861de43c37911a28 in term 0.
I20260619 12:35:27.918196 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfaf38b7edcc46cab4e0dbc832186c18" candidate_uuid: "fb05a9d3c2b94b2d861de43c37911a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf458c0235e462a9ac16b672793b8d0" is_pre_election: true
I20260619 12:35:27.918220 2213 leader_election.cc:302] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ace8d9dcdacd4dbc871e6a0a058997a5, fb05a9d3c2b94b2d861de43c37911a28; no voters:
I20260619 12:35:27.918349 2562 raft_consensus.cc:2454] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb05a9d3c2b94b2d861de43c37911a28 in term 0.
I20260619 12:35:27.918519 2639 raft_consensus.cc:2781] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:27.918581 2639 raft_consensus.cc:482] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:27.918622 2639 raft_consensus.cc:3037] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:27.920343 2639 raft_consensus.cc:504] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.920760 2639 leader_election.cc:288] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [CANDIDATE]: Term 1 election: Requested vote from peers ace8d9dcdacd4dbc871e6a0a058997a5 (127.0.98.2:33813), ccf458c0235e462a9ac16b672793b8d0 (127.0.98.3:46595)
I20260619 12:35:27.921267 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfaf38b7edcc46cab4e0dbc832186c18" candidate_uuid: "fb05a9d3c2b94b2d861de43c37911a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5"
I20260619 12:35:27.921401 2422 raft_consensus.cc:3037] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:27.921666 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfaf38b7edcc46cab4e0dbc832186c18" candidate_uuid: "fb05a9d3c2b94b2d861de43c37911a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf458c0235e462a9ac16b672793b8d0"
I20260619 12:35:27.921808 2562 raft_consensus.cc:3037] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:27.922678 2562 raft_consensus.cc:2454] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb05a9d3c2b94b2d861de43c37911a28 in term 1.
I20260619 12:35:27.922714 2422 raft_consensus.cc:2454] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb05a9d3c2b94b2d861de43c37911a28 in term 1.
I20260619 12:35:27.922994 2213 leader_election.cc:302] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccf458c0235e462a9ac16b672793b8d0, fb05a9d3c2b94b2d861de43c37911a28; no voters:
I20260619 12:35:27.923305 2639 raft_consensus.cc:2781] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:27.923570 2639 raft_consensus.cc:686] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [term 1 LEADER]: Becoming Leader. State: Replica: fb05a9d3c2b94b2d861de43c37911a28, State: Running, Role: LEADER
I20260619 12:35:27.923859 2639 consensus_queue.cc:237] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:27.926496 2125 catalog_manager.cc:5495] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb05a9d3c2b94b2d861de43c37911a28 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fb05a9d3c2b94b2d861de43c37911a28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } health_report { overall_health: UNKNOWN } } }
I20260619 12:35:27.948308 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.944461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35790 (local address 127.0.98.1:41107)
0619 12:35:27.944566 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.944571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.944593 (+ 22us) server_negotiation.cc:407] Connection header received
0619 12:35:27.944715 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.944717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.944769 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.944861 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:27.945097 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.945583 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.946728 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.946976 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.947108 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.947403 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:27.947877 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:27.948042 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.948100 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.948160 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 12:35:27.958314 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.955312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58355 (local address 127.0.98.2:33813)
0619 12:35:27.955451 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.955456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.955664 (+ 208us) server_negotiation.cc:407] Connection header received
0619 12:35:27.955769 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.955771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.955814 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.955878 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:27.956095 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.956593 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.957378 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.957648 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.957724 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.957960 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:27.958028 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:27.958084 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.958160 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.958207 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 12:35:27.958330 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.955247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33813 (local address 127.0.98.1:58355)
0619 12:35:27.955586 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:27.955598 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:27.955687 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:27.955897 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:27.955899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:27.955905 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:35:27.956009 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.956013 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.956684 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.956686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.957285 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.957292 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.957741 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.957743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.957810 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.957850 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:35:27.958099 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:35:27.958103 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:35:27.958149 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 12:35:27.958207 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":106,"threads_started":1}
I20260619 12:35:27.964190 2562 raft_consensus.cc:1261] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Refusing update from remote peer fb05a9d3c2b94b2d861de43c37911a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:27.964426 2422 raft_consensus.cc:1261] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Refusing update from remote peer fb05a9d3c2b94b2d861de43c37911a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:27.964627 2639 consensus_queue.cc:1035] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:27.964924 2681 consensus_queue.cc:1035] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:27.986449 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.982445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41107 (local address 127.0.98.2:57297)
0619 12:35:27.982647 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:27.982664 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:27.982752 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:27.983134 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:27.983137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:27.983146 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:35:27.983251 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.983256 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.984072 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.984075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.984929 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:27.984939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.985733 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.985737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:27.985823 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.985854 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:35:27.986165 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:35:27.986170 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:35:27.986228 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 12:35:27.986316 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 12:35:27.986901 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:27.982505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57297 (local address 127.0.98.1:41107)
0619 12:35:27.982612 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:27.982617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:27.982837 (+ 220us) server_negotiation.cc:407] Connection header received
0619 12:35:27.982932 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:27.982935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:27.982991 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:27.983068 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:27.983343 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.983981 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.985067 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:27.985365 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:27.985475 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:27.985994 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:27.986077 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:27.986155 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:27.986719 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 12:35:27.986773 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 12:35:27.987563 2658 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 12:35:29.553411 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:29.553573 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:29.556177 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc: Sending DeleteTablet for 3 replicas of tablet bfaf38b7edcc46cab4e0dbc832186c18
I20260619 12:35:29.556823 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet bfaf38b7edcc46cab4e0dbc832186c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35788
I20260619 12:35:29.556819 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet bfaf38b7edcc46cab4e0dbc832186c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39922
I20260619 12:35:29.556931 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet bfaf38b7edcc46cab4e0dbc832186c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59886
I20260619 12:35:29.557181 2713 tablet_replica.cc:330] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28: stopping tablet replica
I20260619 12:35:29.557181 2714 tablet_replica.cc:330] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5: stopping tablet replica
I20260619 12:35:29.557317 2714 raft_consensus.cc:2229] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:29.557317 2713 raft_consensus.cc:2229] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:35:29.557446 2714 raft_consensus.cc:2258] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:29.557439 2715 tablet_replica.cc:330] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0: stopping tablet replica
I20260619 12:35:29.557765 2715 raft_consensus.cc:2229] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:29.557919 2713 raft_consensus.cc:2258] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:29.557965 2715 raft_consensus.cc:2258] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:29.558738 2714 ts_tablet_manager.cc:1905] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:29.558820 2715 ts_tablet_manager.cc:1905] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:29.560931 2713 ts_tablet_manager.cc:1905] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:29.561967 2714 ts_tablet_manager.cc:1918] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 12:35:29.562057 2714 log.cc:1196] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/bfaf38b7edcc46cab4e0dbc832186c18
I20260619 12:35:29.562121 2715 ts_tablet_manager.cc:1918] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 12:35:29.562204 2715 log.cc:1196] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/bfaf38b7edcc46cab4e0dbc832186c18
I20260619 12:35:29.562364 2714 ts_tablet_manager.cc:1939] T bfaf38b7edcc46cab4e0dbc832186c18 P ace8d9dcdacd4dbc871e6a0a058997a5: Deleting consensus metadata
I20260619 12:35:29.562489 2715 ts_tablet_manager.cc:1939] T bfaf38b7edcc46cab4e0dbc832186c18 P ccf458c0235e462a9ac16b672793b8d0: Deleting consensus metadata
I20260619 12:35:29.563252 2109 catalog_manager.cc:4841] TS ace8d9dcdacd4dbc871e6a0a058997a5 (127.0.98.2:33813): tablet bfaf38b7edcc46cab4e0dbc832186c18 (table test-table [id=0d938ca41845425a9442a8eb2b1176a6]) successfully deleted
I20260619 12:35:29.563503 2109 catalog_manager.cc:4841] TS ccf458c0235e462a9ac16b672793b8d0 (127.0.98.3:46595): tablet bfaf38b7edcc46cab4e0dbc832186c18 (table test-table [id=0d938ca41845425a9442a8eb2b1176a6]) successfully deleted
I20260619 12:35:29.563788 2713 ts_tablet_manager.cc:1918] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 12:35:29.563853 2713 log.cc:1196] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/wal/wals/bfaf38b7edcc46cab4e0dbc832186c18
I20260619 12:35:29.564127 2713 ts_tablet_manager.cc:1939] T bfaf38b7edcc46cab4e0dbc832186c18 P fb05a9d3c2b94b2d861de43c37911a28: Deleting consensus metadata
I20260619 12:35:29.565176 2112 catalog_manager.cc:4841] TS fb05a9d3c2b94b2d861de43c37911a28 (127.0.98.1:41107): tablet bfaf38b7edcc46cab4e0dbc832186c18 (table test-table [id=0d938ca41845425a9442a8eb2b1176a6]) successfully deleted
Jun 19 12:35:29 dist-test-slave-kdrb krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872529, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 12:35:29 dist-test-slave-kdrb krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872529, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:29.587257 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:29.577497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55078 (local address 127.0.98.62:36041)
0619 12:35:29.577787 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:29.577798 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:29.577893 (+ 95us) server_negotiation.cc:407] Connection header received
0619 12:35:29.578083 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:29.578086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:29.578136 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:29.578232 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:29.578957 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.579575 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.580500 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.580827 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.580958 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:29.583693 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:29.583697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:29.583700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:29.583736 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:29.585394 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:29.585853 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:29.585857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:29.585859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:29.585954 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:29.586240 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:29.586247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:29.586249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:29.586531 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:29.586678 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:29.586939 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 12:35:29.587084 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":106,"threads_started":1}
I20260619 12:35:29.590189 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:35:29.590688 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:29.595427 2262 tablet_service.cc:1467] Processing CreateTablet for tablet a62a062a4afc44c3a32a072588dd7dba (DEFAULT_TABLE table=test-table [id=a0a5e13460874f8487cbb9ba7b4344ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:29.595425 2401 tablet_service.cc:1467] Processing CreateTablet for tablet a62a062a4afc44c3a32a072588dd7dba (DEFAULT_TABLE table=test-table [id=a0a5e13460874f8487cbb9ba7b4344ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:29.595626 2542 tablet_service.cc:1467] Processing CreateTablet for tablet a62a062a4afc44c3a32a072588dd7dba (DEFAULT_TABLE table=test-table [id=a0a5e13460874f8487cbb9ba7b4344ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:29.595782 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62a062a4afc44c3a32a072588dd7dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:29.595788 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62a062a4afc44c3a32a072588dd7dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:29.595904 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62a062a4afc44c3a32a072588dd7dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:29.598081 2724 tablet_bootstrap.cc:492] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28: Bootstrap starting.
I20260619 12:35:29.598817 2725 tablet_bootstrap.cc:492] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0: Bootstrap starting.
I20260619 12:35:29.599062 2724 tablet_bootstrap.cc:654] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:29.599671 2726 tablet_bootstrap.cc:492] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5: Bootstrap starting.
I20260619 12:35:29.600173 2725 tablet_bootstrap.cc:654] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:29.600443 2724 tablet_bootstrap.cc:492] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28: No bootstrap required, opened a new log
I20260619 12:35:29.600589 2726 tablet_bootstrap.cc:654] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:29.600587 2724 ts_tablet_manager.cc:1397] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 12:35:29.601068 2724 raft_consensus.cc:348] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.601209 2724 raft_consensus.cc:374] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:29.601251 2724 raft_consensus.cc:729] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb05a9d3c2b94b2d861de43c37911a28, State: Initialized, Role: FOLLOWER
I20260619 12:35:29.601377 2725 tablet_bootstrap.cc:492] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0: No bootstrap required, opened a new log
I20260619 12:35:29.601402 2724 consensus_queue.cc:260] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.601462 2725 ts_tablet_manager.cc:1397] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 12:35:29.601687 2726 tablet_bootstrap.cc:492] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5: No bootstrap required, opened a new log
I20260619 12:35:29.601686 2724 ts_tablet_manager.cc:1428] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 12:35:29.601763 2726 ts_tablet_manager.cc:1397] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:35:29.601969 2725 raft_consensus.cc:348] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.602082 2725 raft_consensus.cc:374] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:29.602110 2725 raft_consensus.cc:729] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf458c0235e462a9ac16b672793b8d0, State: Initialized, Role: FOLLOWER
I20260619 12:35:29.602118 2726 raft_consensus.cc:348] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.602238 2726 raft_consensus.cc:374] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:29.602275 2726 raft_consensus.cc:729] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace8d9dcdacd4dbc871e6a0a058997a5, State: Initialized, Role: FOLLOWER
I20260619 12:35:29.602238 2725 consensus_queue.cc:260] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.602419 2726 consensus_queue.cc:260] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.602603 2725 ts_tablet_manager.cc:1428] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:35:29.602679 2726 ts_tablet_manager.cc:1428] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:35:29.609678 2730 raft_consensus.cc:482] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:29.609820 2730 raft_consensus.cc:504] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.610917 2730 leader_election.cc:288] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb05a9d3c2b94b2d861de43c37911a28 (127.0.98.1:41107), ace8d9dcdacd4dbc871e6a0a058997a5 (127.0.98.2:33813)
I20260619 12:35:29.616070 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:29.611365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53557 (local address 127.0.98.1:41107)
0619 12:35:29.611701 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:29.611710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:29.611728 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:35:29.611823 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:29.611826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:29.611905 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:29.611990 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:29.612369 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.613199 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.614749 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.615410 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.615530 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:29.615586 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:29.615842 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 12:35:29.615927 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":182,"threads_started":1}
I20260619 12:35:29.616070 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:29.611263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48497 (local address 127.0.98.2:33813)
0619 12:35:29.611737 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:29.611753 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:29.611765 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:35:29.611821 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:29.611823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:29.611874 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:29.611985 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:29.612369 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.613192 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.614695 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.615374 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.615519 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:29.615587 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:29.615842 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 12:35:29.615924 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":107,"threads_started":1}
I20260619 12:35:29.616132 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:29.611156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33813 (local address 127.0.98.3:48497)
0619 12:35:29.611612 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:29.611626 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:29.611735 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:29.612017 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:29.612021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:29.612046 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:29.612255 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:29.612261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.613333 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.613336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:29.614570 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:29.614580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.615545 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.615549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:29.615654 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:29.615707 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 12:35:29.615791 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 12:35:29.615910 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":129,"spinlock_wait_cycles":27520,"thread_start_us":98,"threads_started":1}
I20260619 12:35:29.616138 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:29.611156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41107 (local address 127.0.98.3:53557)
0619 12:35:29.611591 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:29.611607 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:29.611735 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:29.612018 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:29.612021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:29.612046 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:29.612247 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:29.612254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.613324 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.613329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:29.614579 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:29.614590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.615589 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.615592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:29.615710 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:29.615726 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 12:35:29.615791 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 12:35:29.615910 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":259,"spinlock_wait_cycles":15360,"thread_start_us":124,"threads_started":1}
I20260619 12:35:29.616820 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62a062a4afc44c3a32a072588dd7dba" candidate_uuid: "ccf458c0235e462a9ac16b672793b8d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" is_pre_election: true
I20260619 12:35:29.616884 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62a062a4afc44c3a32a072588dd7dba" candidate_uuid: "ccf458c0235e462a9ac16b672793b8d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb05a9d3c2b94b2d861de43c37911a28" is_pre_election: true
I20260619 12:35:29.617007 2422 raft_consensus.cc:2454] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf458c0235e462a9ac16b672793b8d0 in term 0.
I20260619 12:35:29.617059 2282 raft_consensus.cc:2454] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf458c0235e462a9ac16b672793b8d0 in term 0.
I20260619 12:35:29.617403 2493 leader_election.cc:302] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: ace8d9dcdacd4dbc871e6a0a058997a5, ccf458c0235e462a9ac16b672793b8d0, fb05a9d3c2b94b2d861de43c37911a28; no voters:
I20260619 12:35:29.617686 2730 raft_consensus.cc:2781] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:29.617751 2730 raft_consensus.cc:482] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:29.617779 2730 raft_consensus.cc:3037] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:29.618763 2730 raft_consensus.cc:504] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.619169 2730 leader_election.cc:288] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [CANDIDATE]: Term 1 election: Requested vote from peers fb05a9d3c2b94b2d861de43c37911a28 (127.0.98.1:41107), ace8d9dcdacd4dbc871e6a0a058997a5 (127.0.98.2:33813)
I20260619 12:35:29.619498 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62a062a4afc44c3a32a072588dd7dba" candidate_uuid: "ccf458c0235e462a9ac16b672793b8d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb05a9d3c2b94b2d861de43c37911a28"
I20260619 12:35:29.619557 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62a062a4afc44c3a32a072588dd7dba" candidate_uuid: "ccf458c0235e462a9ac16b672793b8d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5"
I20260619 12:35:29.619656 2282 raft_consensus.cc:3037] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:29.619706 2422 raft_consensus.cc:3037] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:29.620702 2282 raft_consensus.cc:2454] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf458c0235e462a9ac16b672793b8d0 in term 1.
I20260619 12:35:29.620723 2422 raft_consensus.cc:2454] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf458c0235e462a9ac16b672793b8d0 in term 1.
I20260619 12:35:29.621001 2496 leader_election.cc:302] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccf458c0235e462a9ac16b672793b8d0, fb05a9d3c2b94b2d861de43c37911a28; no voters:
I20260619 12:35:29.621191 2730 raft_consensus.cc:2781] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:29.621426 2730 raft_consensus.cc:686] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [term 1 LEADER]: Becoming Leader. State: Replica: ccf458c0235e462a9ac16b672793b8d0, State: Running, Role: LEADER
I20260619 12:35:29.621690 2730 consensus_queue.cc:237] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.624698 2124 catalog_manager.cc:5495] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccf458c0235e462a9ac16b672793b8d0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ccf458c0235e462a9ac16b672793b8d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } health_report { overall_health: HEALTHY } } }
I20260619 12:35:29.641077 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:29.637083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59900 (local address 127.0.98.3:46595)
0619 12:35:29.637349 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:29.637356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:29.637373 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:35:29.637446 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:29.637448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:29.637498 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:29.637605 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:29.637892 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.638476 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.639416 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.639735 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.639820 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:29.640128 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:29.640611 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:29.640701 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:29.640868 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 12:35:29.640937 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":128,"threads_started":1}
I20260619 12:35:29.652175 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:29.648652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35529 (local address 127.0.98.2:33813)
0619 12:35:29.648790 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:29.648795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:29.648895 (+ 100us) server_negotiation.cc:407] Connection header received
0619 12:35:29.649070 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:29.649073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:29.649149 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:29.649235 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:29.649508 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.650033 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.650967 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.651198 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.651323 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:29.651687 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:29.651769 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:29.651853 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:29.651949 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 12:35:29.652025 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 12:35:29.652189 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:29.648396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33813 (local address 127.0.98.3:35529)
0619 12:35:29.648757 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:29.648812 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:29.648963 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:29.649247 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:29.649250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:29.649258 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:35:29.649400 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:29.649406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.650154 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.650159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:29.650838 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:29.650848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.651325 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.651329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:29.651433 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:29.651519 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:35:29.651858 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:35:29.651867 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 12:35:29.651930 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 12:35:29.652022 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":129,"threads_started":1}
I20260619 12:35:29.659142 2422 raft_consensus.cc:1261] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Refusing update from remote peer ccf458c0235e462a9ac16b672793b8d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:29.659392 2282 raft_consensus.cc:1261] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28 [term 1 FOLLOWER]: Refusing update from remote peer ccf458c0235e462a9ac16b672793b8d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:29.659821 2735 consensus_queue.cc:1035] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:29.660025 2730 consensus_queue.cc:1035] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:29.679185 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:29.675277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46595 (local address 127.0.98.2:37561)
0619 12:35:29.675646 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:29.675664 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:29.675779 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:29.676065 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:29.676069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:29.676077 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:35:29.676192 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:29.676203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.677129 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.677133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:29.677760 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:29.677769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.678395 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.678399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:29.678509 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:29.678544 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:35:29.678870 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:35:29.678879 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 12:35:29.678945 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 12:35:29.679040 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":143,"threads_started":1}
I20260619 12:35:29.679164 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:29.675427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37561 (local address 127.0.98.3:46595)
0619 12:35:29.675592 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:29.675597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:29.675752 (+ 155us) server_negotiation.cc:407] Connection header received
0619 12:35:29.675875 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:29.675878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:29.675933 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:29.676039 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:29.676310 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.677014 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.677909 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.678154 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.678397 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:29.678689 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:29.678776 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:29.678863 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:29.678955 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 12:35:29.679015 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 12:35:29.702733 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:29.702903 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:29.705273 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc: Sending DeleteTablet for 3 replicas of tablet a62a062a4afc44c3a32a072588dd7dba
I20260619 12:35:29.705947 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet a62a062a4afc44c3a32a072588dd7dba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35788
I20260619 12:35:29.705984 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet a62a062a4afc44c3a32a072588dd7dba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59886
I20260619 12:35:29.706084 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet a62a062a4afc44c3a32a072588dd7dba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39922
I20260619 12:35:29.706152 2715 tablet_replica.cc:330] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0: stopping tablet replica
I20260619 12:35:29.706290 2714 tablet_replica.cc:330] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5: stopping tablet replica
I20260619 12:35:29.706413 2714 raft_consensus.cc:2229] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:29.706499 2713 tablet_replica.cc:330] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28: stopping tablet replica
I20260619 12:35:29.706555 2714 raft_consensus.cc:2258] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:29.706633 2713 raft_consensus.cc:2229] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:29.706761 2713 raft_consensus.cc:2258] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:29.706981 2715 raft_consensus.cc:2229] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:35:29.707370 2715 raft_consensus.cc:2258] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:29.707597 2714 ts_tablet_manager.cc:1905] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:29.708138 2715 ts_tablet_manager.cc:1905] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:29.710276 2715 ts_tablet_manager.cc:1918] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:35:29.710393 2715 log.cc:1196] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/a62a062a4afc44c3a32a072588dd7dba
I20260619 12:35:29.710455 2714 ts_tablet_manager.cc:1918] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:35:29.710548 2714 log.cc:1196] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/a62a062a4afc44c3a32a072588dd7dba
I20260619 12:35:29.710660 2715 ts_tablet_manager.cc:1939] T a62a062a4afc44c3a32a072588dd7dba P ccf458c0235e462a9ac16b672793b8d0: Deleting consensus metadata
I20260619 12:35:29.710819 2714 ts_tablet_manager.cc:1939] T a62a062a4afc44c3a32a072588dd7dba P ace8d9dcdacd4dbc871e6a0a058997a5: Deleting consensus metadata
I20260619 12:35:29.711427 2713 ts_tablet_manager.cc:1905] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:29.711818 2109 catalog_manager.cc:4841] TS ccf458c0235e462a9ac16b672793b8d0 (127.0.98.3:46595): tablet a62a062a4afc44c3a32a072588dd7dba (table test-table [id=a0a5e13460874f8487cbb9ba7b4344ff]) successfully deleted
I20260619 12:35:29.712244 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:35:29.712723 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:29.712759 2109 catalog_manager.cc:4841] TS ace8d9dcdacd4dbc871e6a0a058997a5 (127.0.98.2:33813): tablet a62a062a4afc44c3a32a072588dd7dba (table test-table [id=a0a5e13460874f8487cbb9ba7b4344ff]) successfully deleted
I20260619 12:35:29.718077 2262 tablet_service.cc:1467] Processing CreateTablet for tablet a6d58713f8e643c0aa3c258304bc5dfa (DEFAULT_TABLE table=test-table [id=423227da4aab4197a2918eab985f3d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:29.718266 2401 tablet_service.cc:1467] Processing CreateTablet for tablet a6d58713f8e643c0aa3c258304bc5dfa (DEFAULT_TABLE table=test-table [id=423227da4aab4197a2918eab985f3d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:29.718396 2542 tablet_service.cc:1467] Processing CreateTablet for tablet a6d58713f8e643c0aa3c258304bc5dfa (DEFAULT_TABLE table=test-table [id=423227da4aab4197a2918eab985f3d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:29.718521 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6d58713f8e643c0aa3c258304bc5dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:29.718663 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6d58713f8e643c0aa3c258304bc5dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:29.721065 2725 tablet_bootstrap.cc:492] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0: Bootstrap starting.
I20260619 12:35:29.721637 2726 tablet_bootstrap.cc:492] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5: Bootstrap starting.
I20260619 12:35:29.722155 2725 tablet_bootstrap.cc:654] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:29.722548 2726 tablet_bootstrap.cc:654] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:29.723012 2713 ts_tablet_manager.cc:1918] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:35:29.723109 2713 log.cc:1196] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-0/wal/wals/a62a062a4afc44c3a32a072588dd7dba
I20260619 12:35:29.723388 2713 ts_tablet_manager.cc:1939] T a62a062a4afc44c3a32a072588dd7dba P fb05a9d3c2b94b2d861de43c37911a28: Deleting consensus metadata
I20260619 12:35:29.723647 2725 tablet_bootstrap.cc:492] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0: No bootstrap required, opened a new log
I20260619 12:35:29.723652 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6d58713f8e643c0aa3c258304bc5dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:29.723722 2726 tablet_bootstrap.cc:492] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5: No bootstrap required, opened a new log
I20260619 12:35:29.723728 2725 ts_tablet_manager.cc:1397] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 12:35:29.723798 2726 ts_tablet_manager.cc:1397] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:35:29.724207 2726 raft_consensus.cc:348] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.724202 2725 raft_consensus.cc:348] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.724314 2726 raft_consensus.cc:374] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:29.724316 2725 raft_consensus.cc:374] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:29.724344 2726 raft_consensus.cc:729] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace8d9dcdacd4dbc871e6a0a058997a5, State: Initialized, Role: FOLLOWER
I20260619 12:35:29.724349 2725 raft_consensus.cc:729] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf458c0235e462a9ac16b672793b8d0, State: Initialized, Role: FOLLOWER
I20260619 12:35:29.724474 2726 consensus_queue.cc:260] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.724469 2725 consensus_queue.cc:260] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.724717 2726 ts_tablet_manager.cc:1428] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:35:29.726292 2724 tablet_bootstrap.cc:492] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28: Bootstrap starting.
I20260619 12:35:29.726639 2725 ts_tablet_manager.cc:1428] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260619 12:35:29.727231 2724 tablet_bootstrap.cc:654] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:29.728637 2724 tablet_bootstrap.cc:492] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28: No bootstrap required, opened a new log
I20260619 12:35:29.728740 2724 ts_tablet_manager.cc:1397] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 12:35:29.729108 2724 raft_consensus.cc:348] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.729208 2724 raft_consensus.cc:374] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:29.729231 2724 raft_consensus.cc:729] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb05a9d3c2b94b2d861de43c37911a28, State: Initialized, Role: FOLLOWER
I20260619 12:35:29.729393 2724 consensus_queue.cc:260] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.729668 2724 ts_tablet_manager.cc:1428] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 12:35:29.735126 2112 catalog_manager.cc:4841] TS fb05a9d3c2b94b2d861de43c37911a28 (127.0.98.1:41107): tablet a62a062a4afc44c3a32a072588dd7dba (table test-table [id=a0a5e13460874f8487cbb9ba7b4344ff]) successfully deleted
I20260619 12:35:29.819104 2705 raft_consensus.cc:482] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:29.819288 2705 raft_consensus.cc:504] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.819825 2705 leader_election.cc:288] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ace8d9dcdacd4dbc871e6a0a058997a5 (127.0.98.2:33813), ccf458c0235e462a9ac16b672793b8d0 (127.0.98.3:46595)
I20260619 12:35:29.820325 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6d58713f8e643c0aa3c258304bc5dfa" candidate_uuid: "fb05a9d3c2b94b2d861de43c37911a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" is_pre_election: true
I20260619 12:35:29.820497 2422 raft_consensus.cc:2454] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb05a9d3c2b94b2d861de43c37911a28 in term 0.
I20260619 12:35:29.820773 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6d58713f8e643c0aa3c258304bc5dfa" candidate_uuid: "fb05a9d3c2b94b2d861de43c37911a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf458c0235e462a9ac16b672793b8d0" is_pre_election: true
I20260619 12:35:29.820839 2213 leader_election.cc:302] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ace8d9dcdacd4dbc871e6a0a058997a5, fb05a9d3c2b94b2d861de43c37911a28; no voters:
I20260619 12:35:29.820930 2562 raft_consensus.cc:2454] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb05a9d3c2b94b2d861de43c37911a28 in term 0.
I20260619 12:35:29.821065 2705 raft_consensus.cc:2781] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:29.821118 2705 raft_consensus.cc:482] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:29.821167 2705 raft_consensus.cc:3037] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:29.835829 2705 raft_consensus.cc:504] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.836455 2705 leader_election.cc:288] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [CANDIDATE]: Term 1 election: Requested vote from peers ace8d9dcdacd4dbc871e6a0a058997a5 (127.0.98.2:33813), ccf458c0235e462a9ac16b672793b8d0 (127.0.98.3:46595)
I20260619 12:35:29.836975 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6d58713f8e643c0aa3c258304bc5dfa" candidate_uuid: "fb05a9d3c2b94b2d861de43c37911a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5"
I20260619 12:35:29.837139 2422 raft_consensus.cc:3037] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:29.837087 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6d58713f8e643c0aa3c258304bc5dfa" candidate_uuid: "fb05a9d3c2b94b2d861de43c37911a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf458c0235e462a9ac16b672793b8d0"
I20260619 12:35:29.837244 2562 raft_consensus.cc:3037] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:29.838583 2562 raft_consensus.cc:2454] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb05a9d3c2b94b2d861de43c37911a28 in term 1.
I20260619 12:35:29.839093 2422 raft_consensus.cc:2454] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb05a9d3c2b94b2d861de43c37911a28 in term 1.
I20260619 12:35:29.839188 2213 leader_election.cc:302] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccf458c0235e462a9ac16b672793b8d0, fb05a9d3c2b94b2d861de43c37911a28; no voters:
I20260619 12:35:29.839521 2705 raft_consensus.cc:2781] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:29.839633 2705 raft_consensus.cc:686] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [term 1 LEADER]: Becoming Leader. State: Replica: fb05a9d3c2b94b2d861de43c37911a28, State: Running, Role: LEADER
I20260619 12:35:29.839813 2705 consensus_queue.cc:237] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } }
I20260619 12:35:29.841898 2124 catalog_manager.cc:5495] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb05a9d3c2b94b2d861de43c37911a28 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fb05a9d3c2b94b2d861de43c37911a28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb05a9d3c2b94b2d861de43c37911a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 } health_report { overall_health: UNKNOWN } } }
I20260619 12:35:29.865373 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:29.862200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35796 (local address 127.0.98.1:41107)
0619 12:35:29.862322 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:29.862327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:29.862342 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:35:29.862483 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:29.862486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:29.862541 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:29.862622 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:29.862884 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.863412 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.864297 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:29.864533 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:29.864643 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:29.864909 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:29.864993 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:29.865070 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:29.865167 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 12:35:29.865220 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 12:35:29.872296 2422 raft_consensus.cc:1261] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Refusing update from remote peer fb05a9d3c2b94b2d861de43c37911a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:29.872318 2562 raft_consensus.cc:1261] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Refusing update from remote peer fb05a9d3c2b94b2d861de43c37911a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:29.872721 2688 consensus_queue.cc:1035] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf458c0235e462a9ac16b672793b8d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:29.872893 2705 consensus_queue.cc:1035] T a6d58713f8e643c0aa3c258304bc5dfa P fb05a9d3c2b94b2d861de43c37911a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ace8d9dcdacd4dbc871e6a0a058997a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 12:35:29.891022 2694 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 12:35:31.447435 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:31.447657 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:31.450197 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04d409edecf7498b913732ddc59f44bc: Sending DeleteTablet for 3 replicas of tablet a6d58713f8e643c0aa3c258304bc5dfa
I20260619 12:35:31.450834 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet a6d58713f8e643c0aa3c258304bc5dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35788
I20260619 12:35:31.450834 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet a6d58713f8e643c0aa3c258304bc5dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39922
I20260619 12:35:31.450999 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet a6d58713f8e643c0aa3c258304bc5dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59886
I20260619 12:35:31.451325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 2197
I20260619 12:35:31.451975 2779 tablet_replica.cc:330] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0: stopping tablet replica
I20260619 12:35:31.452152 2779 raft_consensus.cc:2229] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:31.452281 2779 raft_consensus.cc:2258] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:31.451242 2777 tablet_replica.cc:330] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5: stopping tablet replica
I20260619 12:35:31.452808 2777 raft_consensus.cc:2229] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:31.452991 2777 raft_consensus.cc:2258] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:31.453236 2779 ts_tablet_manager.cc:1905] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:31.453692 2777 ts_tablet_manager.cc:1905] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:31.455981 2779 ts_tablet_manager.cc:1918] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 12:35:31.456080 2779 log.cc:1196] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/a6d58713f8e643c0aa3c258304bc5dfa
I20260619 12:35:31.456171 2777 ts_tablet_manager.cc:1918] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 12:35:31.456235 2777 log.cc:1196] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/a6d58713f8e643c0aa3c258304bc5dfa
I20260619 12:35:31.456363 2779 ts_tablet_manager.cc:1939] T a6d58713f8e643c0aa3c258304bc5dfa P ccf458c0235e462a9ac16b672793b8d0: Deleting consensus metadata
I20260619 12:35:31.456501 2777 ts_tablet_manager.cc:1939] T a6d58713f8e643c0aa3c258304bc5dfa P ace8d9dcdacd4dbc871e6a0a058997a5: Deleting consensus metadata
I20260619 12:35:31.457177 2109 catalog_manager.cc:4841] TS ace8d9dcdacd4dbc871e6a0a058997a5 (127.0.98.2:33813): tablet a6d58713f8e643c0aa3c258304bc5dfa (table test-table [id=423227da4aab4197a2918eab985f3d1f]) successfully deleted
I20260619 12:35:31.458043 2109 catalog_manager.cc:4841] TS ccf458c0235e462a9ac16b672793b8d0 (127.0.98.3:46595): tablet a6d58713f8e643c0aa3c258304bc5dfa (table test-table [id=423227da4aab4197a2918eab985f3d1f]) successfully deleted
W20260619 12:35:31.461879 2112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41107 (error 108)
I20260619 12:35:31.462083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 2337
I20260619 12:35:31.468515 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:31.468087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41107 (local address 127.0.0.1:35800)
0619 12:35:31.468344 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:31.468424 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":111,"threads_started":1}
W20260619 12:35:31.468786 2112 catalog_manager.cc:4584] TS fb05a9d3c2b94b2d861de43c37911a28 (127.0.98.1:41107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6d58713f8e643c0aa3c258304bc5dfa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41107: connect: Connection refused (error 111)
I20260619 12:35:31.471369 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 2477
I20260619 12:35:31.478212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 2093
2026-06-19T12:35:31Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6390 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:35:31 dist-test-slave-kdrb krb5kdc[2784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:35:31 dist-test-slave-kdrb krb5kdc[2784](info): set up 2 sockets
Jun 19 12:35:31 dist-test-slave-kdrb krb5kdc[2784](info): commencing operation
krb5kdc: starting...
W20260619 12:35:32.523463 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:35:32 dist-test-slave-kdrb krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:35:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:35:32.650521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35191
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:32.763935 2797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:32.764254 2797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:32.764333 2797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:32.768343 2797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:32.768443 2797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:32.768484 2797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:32.768522 2797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:32.768560 2797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:32.773911 2797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:32.775060 2797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:32.776129 2797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:32.787292 2803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:32.787343 2802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:32.787338 2797 server_base.cc:1034] running on GCE node
W20260619 12:35:32.787483 2805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:32.788259 2797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:32.789548 2797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:32.790742 2797 hybrid_clock.cc:648] HybridClock initialized: now 1781872532790712 us; error 49 us; skew 500 ppm
Jun 19 12:35:32 dist-test-slave-kdrb krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:32.793952 2797 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:35:32.795099 2797 webserver.cc:460] Webserver started at http://127.0.98.62:32911/ using document root <none> and password file <none>
I20260619 12:35:32.795786 2797 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:32.795867 2797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:32.796105 2797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:32.798398 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "3940438f1b2044ff8a1fc451593b8f7c"
format_stamp: "Formatted at 2026-06-19 12:35:32 on dist-test-slave-kdrb"
server_key: "378cada08c9657aca0656f85965e49d4"
server_key_iv: "f128f25f20975fa5419851396763f3bb"
server_key_version: "encryptionkey@0"
I20260619 12:35:32.799053 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "3940438f1b2044ff8a1fc451593b8f7c"
format_stamp: "Formatted at 2026-06-19 12:35:32 on dist-test-slave-kdrb"
server_key: "378cada08c9657aca0656f85965e49d4"
server_key_iv: "f128f25f20975fa5419851396763f3bb"
server_key_version: "encryptionkey@0"
I20260619 12:35:32.803364 2797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 12:35:32.806301 2812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:32.808014 2797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:35:32.808187 2797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "3940438f1b2044ff8a1fc451593b8f7c"
format_stamp: "Formatted at 2026-06-19 12:35:32 on dist-test-slave-kdrb"
server_key: "378cada08c9657aca0656f85965e49d4"
server_key_iv: "f128f25f20975fa5419851396763f3bb"
server_key_version: "encryptionkey@0"
I20260619 12:35:32.808324 2797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:32.818773 2797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:32.821916 2797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:32.822139 2797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:32.829443 2797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35191
I20260619 12:35:32.829469 2864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35191 every 8 connection(s)
I20260619 12:35:32.830449 2797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:32.833350 2865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:32.837334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 2797
I20260619 12:35:32.837456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:32.837736 392 external_mini_cluster.cc:1428] Setting key 1da6878aa6bc7d868a4f45afbc7463fe
I20260619 12:35:32.839651 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c: Bootstrap starting.
I20260619 12:35:32.842037 2865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:32.842787 2865 log.cc:824] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:32.844702 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c: No bootstrap required, opened a new log
Jun 19 12:35:32 dist-test-slave-kdrb krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:32.849076 2865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3940438f1b2044ff8a1fc451593b8f7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35191 } }
I20260619 12:35:32.849299 2865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:32.849335 2865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3940438f1b2044ff8a1fc451593b8f7c, State: Initialized, Role: FOLLOWER
I20260619 12:35:32.849854 2865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3940438f1b2044ff8a1fc451593b8f7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35191 } }
I20260619 12:35:32.849979 2865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:32.850061 2865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:32.850183 2865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:32.851089 2865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3940438f1b2044ff8a1fc451593b8f7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35191 } }
I20260619 12:35:32.851532 2865 leader_election.cc:302] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3940438f1b2044ff8a1fc451593b8f7c; no voters:
I20260619 12:35:32.851864 2865 leader_election.cc:288] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:32.851986 2870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:32.852193 2870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [term 1 LEADER]: Becoming Leader. State: Replica: 3940438f1b2044ff8a1fc451593b8f7c, State: Running, Role: LEADER
I20260619 12:35:32.852509 2870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3940438f1b2044ff8a1fc451593b8f7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35191 } }
I20260619 12:35:32.852610 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:32.839290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.0.98.62:35191)
0619 12:35:32.839834 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:32.839858 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:32.839894 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:35:32.840582 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:32.840602 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:32.840944 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:32.841316 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:32.842116 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:32.843236 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:32.844166 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:32.844650 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:32.844830 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:32.847572 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:32.847577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:32.847592 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:32.847620 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:32.849309 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:32.849777 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:32.849783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:32.849796 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:32.849908 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:32.850225 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:32.850229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:32.850231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:32.850651 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:32.850867 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:32.851166 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 12:35:32.851527 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":160,"threads_started":1}
I20260619 12:35:32.853204 2865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:32.853708 2870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3940438f1b2044ff8a1fc451593b8f7c. Latest consensus state: current_term: 1 leader_uuid: "3940438f1b2044ff8a1fc451593b8f7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3940438f1b2044ff8a1fc451593b8f7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35191 } } }
I20260619 12:35:32.853812 2870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:32.854139 2871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3940438f1b2044ff8a1fc451593b8f7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3940438f1b2044ff8a1fc451593b8f7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35191 } } }
I20260619 12:35:32.854241 2871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:32.855103 2874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:32.858532 2874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:32.864053 2874 catalog_manager.cc:1347] Generated new cluster ID: 0ed77c2ceb9a4484a28500231fac116c
I20260619 12:35:32.864130 2874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:32.886758 2874 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:32.888015 2874 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:32.899212 2874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3940438f1b2044ff8a1fc451593b8f7c: Generated new TSK 0
I20260619 12:35:32.900041 2874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:35:32.938077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35191
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:33.054132 2891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:33.054414 2891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:33.054461 2891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:33.058652 2891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:33.058730 2891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:33.058792 2891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:35:33.064217 2891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:33.065223 2891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:33.066185 2891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:33.076099 2899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:33.076193 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:33.076198 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:33.076341 2891 server_base.cc:1034] running on GCE node
I20260619 12:35:33.076858 2891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:33.077605 2891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:33.078800 2891 hybrid_clock.cc:648] HybridClock initialized: now 1781872533078761 us; error 52 us; skew 500 ppm
Jun 19 12:35:33 dist-test-slave-kdrb krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:33.081743 2891 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:35:33.082851 2891 webserver.cc:460] Webserver started at http://127.0.98.1:45953/ using document root <none> and password file <none>
I20260619 12:35:33.083573 2891 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:33.083678 2891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:33.083971 2891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:33.086376 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "81b811becd65428aa1dad1215cf15629"
format_stamp: "Formatted at 2026-06-19 12:35:33 on dist-test-slave-kdrb"
server_key: "f5e636e4ea2385ee2dbdbc777279d477"
server_key_iv: "af658e40e033cdf26bb2875a79d1dab5"
server_key_version: "encryptionkey@0"
I20260619 12:35:33.087052 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "81b811becd65428aa1dad1215cf15629"
format_stamp: "Formatted at 2026-06-19 12:35:33 on dist-test-slave-kdrb"
server_key: "f5e636e4ea2385ee2dbdbc777279d477"
server_key_iv: "af658e40e033cdf26bb2875a79d1dab5"
server_key_version: "encryptionkey@0"
I20260619 12:35:33.091415 2891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 12:35:33.094383 2906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:33.096017 2891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:35:33.096177 2891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "81b811becd65428aa1dad1215cf15629"
format_stamp: "Formatted at 2026-06-19 12:35:33 on dist-test-slave-kdrb"
server_key: "f5e636e4ea2385ee2dbdbc777279d477"
server_key_iv: "af658e40e033cdf26bb2875a79d1dab5"
server_key_version: "encryptionkey@0"
I20260619 12:35:33.096311 2891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:33.122889 2891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:33.126612 2891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:33.126844 2891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:33.127579 2891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:33.128463 2891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:33.128509 2891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:33.128551 2891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:33.128566 2891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:33.137799 2891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37195
I20260619 12:35:33.137817 3019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37195 every 8 connection(s)
I20260619 12:35:33.138804 2891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:35:33.145141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 2891
I20260619 12:35:33.145279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:35:33.145555 392 external_mini_cluster.cc:1428] Setting key dfcc1ccec009afc40797965d5853fe5d
Jun 19 12:35:33 dist-test-slave-kdrb krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:33.154867 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:33.140875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39123 (local address 127.0.98.62:35191)
0619 12:35:33.141030 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:33.141035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:33.142015 (+ 980us) server_negotiation.cc:407] Connection header received
0619 12:35:33.143411 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:33.143415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:33.143513 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:33.143596 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:33.144922 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:33.145772 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:33.147339 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:33.147741 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:33.147842 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:33.150684 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:33.150691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:33.150697 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:33.150732 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:33.152449 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:33.153120 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:33.153124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:33.153126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:33.153213 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:33.153573 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:33.153576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:33.153577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:33.153771 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:33.153898 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:33.154553 (+ 655us) server_negotiation.cc:299] Negotiation successful
0619 12:35:33.154689 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:35:33.155936 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:33.141356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35191 (local address 127.0.98.1:39123)
0619 12:35:33.141836 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:33.141882 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:33.143171 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:33.143786 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:33.143794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:33.144204 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:33.144727 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:33.144742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:33.145930 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:33.145944 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:33.147161 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:33.147171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:33.147915 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:33.147922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:33.148205 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:33.148902 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:33.148938 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:33.150483 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:33.152588 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:33.152595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:33.152614 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:33.153010 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:33.153311 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:33.153314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:33.153316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:33.153471 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:33.153913 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:33.153921 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:33.154305 (+ 384us) client_negotiation.cc:769] Sending connection context
0619 12:35:33.154518 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 12:35:33.154877 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":124,"threads_started":1}
I20260619 12:35:33.157311 3020 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35191
I20260619 12:35:33.157544 3020 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:35:33.158216 3020 heartbeater.cc:511] Master 127.0.98.62:35191 requested a full tablet report, sending...
I20260619 12:35:33.160257 2829 ts_manager.cc:194] Registered new tserver with Master: 81b811becd65428aa1dad1215cf15629 (127.0.98.1:37195)
I20260619 12:35:33.162079 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:35:33.174856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35191
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:33.290633 3025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:33.290920 3025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:33.291005 3025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:33.295315 3025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:33.295395 3025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:33.295516 3025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:35:33.301251 3025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:33.302392 3025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:33.303444 3025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:33.313125 3030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:33.313120 3031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:33.313380 3025 server_base.cc:1034] running on GCE node
W20260619 12:35:33.313125 3033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:33.313882 3025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:33.314546 3025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:33.315724 3025 hybrid_clock.cc:648] HybridClock initialized: now 1781872533315707 us; error 32 us; skew 500 ppm
Jun 19 12:35:33 dist-test-slave-kdrb krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:33.318550 3025 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:35:33.319413 3025 webserver.cc:460] Webserver started at http://127.0.98.2:37321/ using document root <none> and password file <none>
I20260619 12:35:33.320010 3025 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:33.320106 3025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:33.320360 3025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:33.322479 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "0bb708621f49479b8b068687da1875b4"
format_stamp: "Formatted at 2026-06-19 12:35:33 on dist-test-slave-kdrb"
server_key: "356fcb20b6e00da5028f3c6ab4ffaa4e"
server_key_iv: "6ceef57d84d603aa5a05f415c67083d2"
server_key_version: "encryptionkey@0"
I20260619 12:35:33.323067 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0bb708621f49479b8b068687da1875b4"
format_stamp: "Formatted at 2026-06-19 12:35:33 on dist-test-slave-kdrb"
server_key: "356fcb20b6e00da5028f3c6ab4ffaa4e"
server_key_iv: "6ceef57d84d603aa5a05f415c67083d2"
server_key_version: "encryptionkey@0"
I20260619 12:35:33.326920 3025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:35:33.329595 3040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:33.330945 3025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:35:33.331096 3025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "0bb708621f49479b8b068687da1875b4"
format_stamp: "Formatted at 2026-06-19 12:35:33 on dist-test-slave-kdrb"
server_key: "356fcb20b6e00da5028f3c6ab4ffaa4e"
server_key_iv: "6ceef57d84d603aa5a05f415c67083d2"
server_key_version: "encryptionkey@0"
I20260619 12:35:33.331226 3025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:33.343313 3025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:33.346525 3025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:33.346748 3025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:33.347350 3025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:33.348299 3025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:33.348373 3025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:33.348453 3025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:33.348497 3025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:33.357883 3025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44127
I20260619 12:35:33.357930 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44127 every 8 connection(s)
I20260619 12:35:33.358840 3025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:35:33.361514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 3025
I20260619 12:35:33.361634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:35:33.361935 392 external_mini_cluster.cc:1428] Setting key 1f45e10a9cca278f28a516409ed58064
Jun 19 12:35:33 dist-test-slave-kdrb krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:35:33.373859 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:33.360787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53495 (local address 127.0.98.62:35191)
0619 12:35:33.360922 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:33.360929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:33.361853 (+ 924us) server_negotiation.cc:407] Connection header received
0619 12:35:33.363605 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:33.363609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:33.363660 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:33.363748 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:33.364988 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:33.365521 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:33.366704 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:33.366934 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:33.367058 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:33.370107 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:33.370111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:33.370113 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:33.370138 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:33.371723 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:33.372268 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:33.372272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:33.372274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:33.372358 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:33.372732 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:33.372737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:33.372739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:33.372935 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:33.373042 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:33.373553 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 12:35:33.373686 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:35:33.374897 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:33.361191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35191 (local address 127.0.98.2:53495)
0619 12:35:33.361679 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:33.361734 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:33.363363 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:33.363932 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:33.363940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:33.364300 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:33.364828 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:33.364847 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:33.365647 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:33.365651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:33.366562 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:33.366571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:33.367132 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:33.367139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:33.367376 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:33.368077 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:33.368103 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:33.369906 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:33.371855 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:33.371861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:33.371899 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:33.372159 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:33.372477 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:33.372482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:33.372484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:33.372636 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:33.373039 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:33.373046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:33.373315 (+ 269us) client_negotiation.cc:769] Sending connection context
0619 12:35:33.373515 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 12:35:33.373852 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":150,"threads_started":1}
I20260619 12:35:33.376331 3154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35191
I20260619 12:35:33.376613 3154 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:33.377226 3154 heartbeater.cc:511] Master 127.0.98.62:35191 requested a full tablet report, sending...
I20260619 12:35:33.378578 2829 ts_manager.cc:194] Registered new tserver with Master: 0bb708621f49479b8b068687da1875b4 (127.0.98.2:44127)
I20260619 12:35:33.379199 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:35:33.390354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35191
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:33.502081 3159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:33.502404 3159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:33.502451 3159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:33.506697 3159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:33.506768 3159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:33.506831 3159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:35:33.512748 3159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:33.513888 3159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:33.514923 3159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:33.524729 3167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:33.524718 3164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:33.524781 3159 server_base.cc:1034] running on GCE node
W20260619 12:35:33.524725 3165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:33.525413 3159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:33.526093 3159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:33.527258 3159 hybrid_clock.cc:648] HybridClock initialized: now 1781872533527239 us; error 32 us; skew 500 ppm
Jun 19 12:35:33 dist-test-slave-kdrb krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:33.530014 3159 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:35:33.530907 3159 webserver.cc:460] Webserver started at http://127.0.98.3:32813/ using document root <none> and password file <none>
I20260619 12:35:33.531456 3159 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:33.531594 3159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:33.531773 3159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:33.533860 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "4d83ca13821e48afad3bfd5ecf684a55"
format_stamp: "Formatted at 2026-06-19 12:35:33 on dist-test-slave-kdrb"
server_key: "09872cd6419a7825fe00deb095fe32e2"
server_key_iv: "2b6a3e5440a283488e1bb61a54d1dfce"
server_key_version: "encryptionkey@0"
I20260619 12:35:33.534502 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4d83ca13821e48afad3bfd5ecf684a55"
format_stamp: "Formatted at 2026-06-19 12:35:33 on dist-test-slave-kdrb"
server_key: "09872cd6419a7825fe00deb095fe32e2"
server_key_iv: "2b6a3e5440a283488e1bb61a54d1dfce"
server_key_version: "encryptionkey@0"
I20260619 12:35:33.538745 3159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 12:35:33.541358 3174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:33.542704 3159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:35:33.542835 3159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "4d83ca13821e48afad3bfd5ecf684a55"
format_stamp: "Formatted at 2026-06-19 12:35:33 on dist-test-slave-kdrb"
server_key: "09872cd6419a7825fe00deb095fe32e2"
server_key_iv: "2b6a3e5440a283488e1bb61a54d1dfce"
server_key_version: "encryptionkey@0"
I20260619 12:35:33.542961 3159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:33.575353 3159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:33.578481 3159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:33.578685 3159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:33.579308 3159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:33.580184 3159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:33.580257 3159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:33.580332 3159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:33.580384 3159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:33.590004 3159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44473
I20260619 12:35:33.590054 3287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44473 every 8 connection(s)
I20260619 12:35:33.590955 3159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:35:33.597292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 3159
I20260619 12:35:33.597410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:35:33.597676 392 external_mini_cluster.cc:1428] Setting key 23ad06fc6bb0520fd42af49abfd418c8
Jun 19 12:35:33 dist-test-slave-kdrb krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:33.605679 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:33.592938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57729 (local address 127.0.98.62:35191)
0619 12:35:33.593093 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:33.593097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:33.594076 (+ 979us) server_negotiation.cc:407] Connection header received
0619 12:35:33.595431 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:33.595434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:33.595544 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:33.595622 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:33.596902 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:33.597691 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:33.599122 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:33.599431 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:33.599592 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:33.602078 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:33.602082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:33.602084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:33.602107 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:33.603737 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:33.604188 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:33.604191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:33.604192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:33.604264 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:33.604560 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:33.604562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:33.604563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:33.604730 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:33.604799 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:33.605427 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 12:35:33.605548 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:35:33.606556 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:33.593424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35191 (local address 127.0.98.3:57729)
0619 12:35:33.593911 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:33.593971 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:33.595214 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:33.595870 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:33.595877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:33.596259 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:33.596761 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:33.596774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:33.597848 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:33.597850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:33.599040 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:33.599050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:33.599660 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:33.599665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:33.599879 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:33.600421 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:33.600446 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:33.601912 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:33.603835 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:33.603839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:33.603852 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:33.604105 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:33.604344 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:33.604346 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:33.604348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:33.604486 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:33.604865 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:33.604870 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:33.605128 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 12:35:33.605325 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 12:35:33.605604 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":110,"threads_started":1}
I20260619 12:35:33.607986 3288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35191
I20260619 12:35:33.608263 3288 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:33.608796 3288 heartbeater.cc:511] Master 127.0.98.62:35191 requested a full tablet report, sending...
I20260619 12:35:33.609902 2829 ts_manager.cc:194] Registered new tserver with Master: 4d83ca13821e48afad3bfd5ecf684a55 (127.0.98.3:44473)
I20260619 12:35:33.610690 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57729
I20260619 12:35:33.611784 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 12:35:33.624776 2868 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:35:33.623201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.0.98.62:35191)
0619 12:35:33.623355 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:33.623360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:33.623635 (+ 275us) server_negotiation.cc:407] Connection header received
0619 12:35:33.623750 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:33.623752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:33.623796 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:33.623867 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:33.624616 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51484: BlockingRecv error: recv got EOF from 127.0.0.1:51484 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 12:35:33.625916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 2891
I20260619 12:35:33.631237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 3025
I20260619 12:35:33.636647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 3159
I20260619 12:35:33.642030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 2797
2026-06-19T12:35:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2162 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:35:33 dist-test-slave-kdrb krb5kdc[3300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:35:33 dist-test-slave-kdrb krb5kdc[3300](info): set up 2 sockets
Jun 19 12:35:33 dist-test-slave-kdrb krb5kdc[3300](info): commencing operation
krb5kdc: starting...
W20260619 12:35:34.687453 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:35:34 dist-test-slave-kdrb krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:35:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:35:34.810400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40159
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:34.922077 3313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:34.922406 3313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:34.922500 3313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:34.926730 3313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:34.926844 3313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:34.926893 3313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:34.926954 3313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:34.926985 3313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:34.932920 3313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:34.934155 3313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:34.935243 3313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:35:34.943837 3313 server_base.cc:1034] running on GCE node
W20260619 12:35:34.943758 3321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:34.943733 3319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:34.943739 3318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:34.944764 3313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:34.945940 3313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:34.947129 3313 hybrid_clock.cc:648] HybridClock initialized: now 1781872534947096 us; error 50 us; skew 500 ppm
Jun 19 12:35:34 dist-test-slave-kdrb krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:34.950011 3313 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:35:34.951085 3313 webserver.cc:460] Webserver started at http://127.0.98.62:44077/ using document root <none> and password file <none>
I20260619 12:35:34.951820 3313 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:34.951923 3313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:34.952157 3313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:34.954245 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "44ff31f9c6e14bf883cdf047f2167223"
format_stamp: "Formatted at 2026-06-19 12:35:34 on dist-test-slave-kdrb"
server_key: "dd0d422e305037f8bb862aae8264007f"
server_key_iv: "f8e4b44b400a78aea7f9773f2942b6c6"
server_key_version: "encryptionkey@0"
I20260619 12:35:34.954829 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "44ff31f9c6e14bf883cdf047f2167223"
format_stamp: "Formatted at 2026-06-19 12:35:34 on dist-test-slave-kdrb"
server_key: "dd0d422e305037f8bb862aae8264007f"
server_key_iv: "f8e4b44b400a78aea7f9773f2942b6c6"
server_key_version: "encryptionkey@0"
I20260619 12:35:34.958585 3313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260619 12:35:34.961107 3328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:34.962332 3313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:35:34.962497 3313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "44ff31f9c6e14bf883cdf047f2167223"
format_stamp: "Formatted at 2026-06-19 12:35:34 on dist-test-slave-kdrb"
server_key: "dd0d422e305037f8bb862aae8264007f"
server_key_iv: "f8e4b44b400a78aea7f9773f2942b6c6"
server_key_version: "encryptionkey@0"
I20260619 12:35:34.962620 3313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:34.988905 3313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:34.992441 3313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:34.992674 3313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:35.000020 3313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40159
I20260619 12:35:35.000335 3380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40159 every 8 connection(s)
I20260619 12:35:35.001083 3313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:35.004189 3381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:35.007027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 3313
I20260619 12:35:35.007158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:35.007454 392 external_mini_cluster.cc:1428] Setting key f72768041a7a1dd291ac0084a84e2a55
I20260619 12:35:35.010721 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223: Bootstrap starting.
I20260619 12:35:35.013458 3381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:35.014288 3381 log.cc:824] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:35.016520 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223: No bootstrap required, opened a new log
Jun 19 12:35:35 dist-test-slave-kdrb krb5kdc[3300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:35.019573 3381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ff31f9c6e14bf883cdf047f2167223" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40159 } }
I20260619 12:35:35.019840 3381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:35.019932 3381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ff31f9c6e14bf883cdf047f2167223, State: Initialized, Role: FOLLOWER
I20260619 12:35:35.020433 3381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ff31f9c6e14bf883cdf047f2167223" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40159 } }
I20260619 12:35:35.020586 3381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:35.020653 3381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:35.020792 3381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:35.021756 3381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ff31f9c6e14bf883cdf047f2167223" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40159 } }
I20260619 12:35:35.022114 3381 leader_election.cc:302] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44ff31f9c6e14bf883cdf047f2167223; no voters:
I20260619 12:35:35.022457 3381 leader_election.cc:288] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:35.022569 3386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:35.022763 3386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [term 1 LEADER]: Becoming Leader. State: Replica: 44ff31f9c6e14bf883cdf047f2167223, State: Running, Role: LEADER
I20260619 12:35:35.022956 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:35.008824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.0.98.62:40159)
0619 12:35:35.009260 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:35.009275 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:35.009311 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:35:35.009961 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:35.009979 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:35.010323 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:35.010618 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:35.011624 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:35.012660 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:35.013821 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:35.014296 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:35.014503 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:35.017224 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:35.017238 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:35.017253 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:35.017280 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:35.019665 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:35.020179 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:35.020184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:35.020189 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:35.020302 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:35.020611 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:35.020615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:35.020616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:35.021149 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:35.021370 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:35.021620 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 12:35:35.021966 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":138,"threads_started":1}
I20260619 12:35:35.023094 3386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ff31f9c6e14bf883cdf047f2167223" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40159 } }
I20260619 12:35:35.023780 3381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:35.024276 3388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44ff31f9c6e14bf883cdf047f2167223. Latest consensus state: current_term: 1 leader_uuid: "44ff31f9c6e14bf883cdf047f2167223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ff31f9c6e14bf883cdf047f2167223" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40159 } } }
I20260619 12:35:35.024428 3388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:35.024824 3390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:35.026371 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:35:35.026446 3387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44ff31f9c6e14bf883cdf047f2167223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ff31f9c6e14bf883cdf047f2167223" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40159 } } }
I20260619 12:35:35.026542 3387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:35.028024 3390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:35.033520 3390 catalog_manager.cc:1347] Generated new cluster ID: 39bdd7dd60684fa69ba5cc986cea11b2
I20260619 12:35:35.033648 3390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:35.047250 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:35.041088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.0.98.62:40159)
0619 12:35:35.041207 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:35.041213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:35.041582 (+ 369us) server_negotiation.cc:407] Connection header received
0619 12:35:35.041697 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:35.041701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:35.041757 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:35.041846 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:35.042407 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:35.043236 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:35.044696 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:35.045079 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:35.045186 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:35.046122 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:35.046126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:35.046129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:35:35.046730 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:35.046772 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:35:35.046792 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:35.046880 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:35.047049 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 12:35:35.047109 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 12:35:35.048467 3390 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:35.049324 3390 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:35.054353 3390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44ff31f9c6e14bf883cdf047f2167223: Generated new TSK 0
I20260619 12:35:35.055022 3390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:35:35.063684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 3313
2026-06-19T12:35:35Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1419 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:35:35 dist-test-slave-kdrb krb5kdc[3414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:35:35 dist-test-slave-kdrb krb5kdc[3414](info): set up 2 sockets
Jun 19 12:35:35 dist-test-slave-kdrb krb5kdc[3414](info): commencing operation
krb5kdc: starting...
W20260619 12:35:36.379683 392 mini_kdc.cc:121] Time spent starting KDC: real 1.296s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:35:36 dist-test-slave-kdrb krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:35:36.518247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46845
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:36.632594 3427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:36.632930 3427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:36.633021 3427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:36.637176 3427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:36.637284 3427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:36.637323 3427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:36.637363 3427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:36.637398 3427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:36.642794 3427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:36.643998 3427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:36.645074 3427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:36.654088 3433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:36.654194 3427 server_base.cc:1034] running on GCE node
W20260619 12:35:36.654119 3432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:36.654139 3435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:36.654850 3427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:36.656098 3427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:36.657279 3427 hybrid_clock.cc:648] HybridClock initialized: now 1781872536657252 us; error 39 us; skew 500 ppm
Jun 19 12:35:36 dist-test-slave-kdrb krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:36.664537 3427 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:35:36.665584 3427 webserver.cc:460] Webserver started at http://127.0.98.62:34549/ using document root <none> and password file <none>
I20260619 12:35:36.666198 3427 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:36.666265 3427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:36.666529 3427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:36.668733 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "b0e76bc41ca54b889c978db781eed92d"
format_stamp: "Formatted at 2026-06-19 12:35:36 on dist-test-slave-kdrb"
server_key: "36673e68cb33a47c42a76550eca4fd8c"
server_key_iv: "83e6922a9ff69c6adfa4b1e55ed67ea6"
server_key_version: "encryptionkey@0"
I20260619 12:35:36.669305 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0e76bc41ca54b889c978db781eed92d"
format_stamp: "Formatted at 2026-06-19 12:35:36 on dist-test-slave-kdrb"
server_key: "36673e68cb33a47c42a76550eca4fd8c"
server_key_iv: "83e6922a9ff69c6adfa4b1e55ed67ea6"
server_key_version: "encryptionkey@0"
I20260619 12:35:36.673059 3427 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:35:36.675451 3442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:36.676762 3427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:35:36.676931 3427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "b0e76bc41ca54b889c978db781eed92d"
format_stamp: "Formatted at 2026-06-19 12:35:36 on dist-test-slave-kdrb"
server_key: "36673e68cb33a47c42a76550eca4fd8c"
server_key_iv: "83e6922a9ff69c6adfa4b1e55ed67ea6"
server_key_version: "encryptionkey@0"
I20260619 12:35:36.677054 3427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:36.688853 3427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:36.691849 3427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:36.692081 3427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:36.699098 3427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46845
I20260619 12:35:36.699362 3494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46845 every 8 connection(s)
I20260619 12:35:36.700138 3427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:36.702848 3495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:36.705173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 3427
I20260619 12:35:36.705295 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:36.705585 392 external_mini_cluster.cc:1428] Setting key 1c4d1442e1198e56688d4f7ac68ed7a6
I20260619 12:35:36.709241 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d: Bootstrap starting.
I20260619 12:35:36.711519 3495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:36.712266 3495 log.cc:824] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:36.714604 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d: No bootstrap required, opened a new log
Jun 19 12:35:36 dist-test-slave-kdrb krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:36.719455 3495 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e76bc41ca54b889c978db781eed92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46845 } }
I20260619 12:35:36.719764 3495 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:36.719820 3495 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e76bc41ca54b889c978db781eed92d, State: Initialized, Role: FOLLOWER
I20260619 12:35:36.720304 3495 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e76bc41ca54b889c978db781eed92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46845 } }
I20260619 12:35:36.720458 3495 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:36.720523 3495 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:36.720616 3495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:36.720614 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:36.706990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.0.98.62:46845)
0619 12:35:36.707439 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:36.707458 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:36.707555 (+ 97us) server_negotiation.cc:407] Connection header received
0619 12:35:36.708395 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:36.708416 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:36.708762 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:36.709121 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:36.709970 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:36.710967 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:36.711966 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:36.712498 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:36.712662 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:36.715412 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:36.715419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:36.715435 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:36.715492 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:36.717522 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:36.717956 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:36.717962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:36.717968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:36.718108 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:36.718358 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:36.718362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:36.718364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:36.718900 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:36.719104 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:36.719289 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 12:35:36.719621 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":144,"threads_started":1}
I20260619 12:35:36.721791 3495 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e76bc41ca54b889c978db781eed92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46845 } }
I20260619 12:35:36.722144 3495 leader_election.cc:302] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0e76bc41ca54b889c978db781eed92d; no voters:
I20260619 12:35:36.722467 3495 leader_election.cc:288] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:36.722572 3500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:36.722769 3500 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [term 1 LEADER]: Becoming Leader. State: Replica: b0e76bc41ca54b889c978db781eed92d, State: Running, Role: LEADER
I20260619 12:35:36.723132 3500 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e76bc41ca54b889c978db781eed92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46845 } }
I20260619 12:35:36.723786 3495 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:36.724301 3502 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0e76bc41ca54b889c978db781eed92d. Latest consensus state: current_term: 1 leader_uuid: "b0e76bc41ca54b889c978db781eed92d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e76bc41ca54b889c978db781eed92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46845 } } }
I20260619 12:35:36.724424 3502 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:36.724766 3501 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0e76bc41ca54b889c978db781eed92d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e76bc41ca54b889c978db781eed92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46845 } } }
I20260619 12:35:36.724910 3505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:36.724946 3501 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:36.728325 3505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:36.734179 3505 catalog_manager.cc:1347] Generated new cluster ID: 020ef952d95e4a798c19de4fbbccce2b
I20260619 12:35:36.734267 3505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:36.762471 3505 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:36.763433 3505 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:36.773335 3505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0e76bc41ca54b889c978db781eed92d: Generated new TSK 0
I20260619 12:35:36.774247 3505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:35:36.808146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46845
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:36.921728 3521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:36.922036 3521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:36.922082 3521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:36.926884 3521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:36.926964 3521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:36.927062 3521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:35:36.932653 3521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:36.933861 3521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:36.934948 3521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:36.944847 3527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:36.945150 3521 server_base.cc:1034] running on GCE node
W20260619 12:35:36.944866 3529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:36.944844 3526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:36.945712 3521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:36.946403 3521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:36.947548 3521 hybrid_clock.cc:648] HybridClock initialized: now 1781872536947516 us; error 48 us; skew 500 ppm
Jun 19 12:35:36 dist-test-slave-kdrb krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:36.950320 3521 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:35:36.951232 3521 webserver.cc:460] Webserver started at http://127.0.98.1:39601/ using document root <none> and password file <none>
I20260619 12:35:36.951869 3521 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:36.951959 3521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:36.952193 3521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:36.954329 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "bd0b9e0ab6ce4d5fb093aa2fbee9d1b5"
format_stamp: "Formatted at 2026-06-19 12:35:36 on dist-test-slave-kdrb"
server_key: "fbdba96965a4a1a835b8b07e7e487d07"
server_key_iv: "0a3b7b8b9a7ea45630f281db13d47cdd"
server_key_version: "encryptionkey@0"
I20260619 12:35:36.954910 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bd0b9e0ab6ce4d5fb093aa2fbee9d1b5"
format_stamp: "Formatted at 2026-06-19 12:35:36 on dist-test-slave-kdrb"
server_key: "fbdba96965a4a1a835b8b07e7e487d07"
server_key_iv: "0a3b7b8b9a7ea45630f281db13d47cdd"
server_key_version: "encryptionkey@0"
I20260619 12:35:36.958755 3521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:35:36.961359 3536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:36.962749 3521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 12:35:36.962904 3521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "bd0b9e0ab6ce4d5fb093aa2fbee9d1b5"
format_stamp: "Formatted at 2026-06-19 12:35:36 on dist-test-slave-kdrb"
server_key: "fbdba96965a4a1a835b8b07e7e487d07"
server_key_iv: "0a3b7b8b9a7ea45630f281db13d47cdd"
server_key_version: "encryptionkey@0"
I20260619 12:35:36.963030 3521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:36.978693 3521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:36.982175 3521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:36.982385 3521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:36.982995 3521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:36.983908 3521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:36.983974 3521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:36.984050 3521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:36.984097 3521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:36.992676 3521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36797
I20260619 12:35:36.992698 3649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36797 every 8 connection(s)
I20260619 12:35:36.993597 3521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:35:36.994438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 3521
I20260619 12:35:36.994582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:35:36.994856 392 external_mini_cluster.cc:1428] Setting key d1f183434f8e8b821f929a545462572d
Jun 19 12:35:37 dist-test-slave-kdrb krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:35:37.010454 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:36.995614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51013 (local address 127.0.98.62:46845)
0619 12:35:36.995747 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:36.995752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:36.996752 (+ 1000us) server_negotiation.cc:407] Connection header received
0619 12:35:36.998518 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:36.998521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:36.998566 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:36.998629 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:36.999988 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:37.000909 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:37.002679 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:37.002945 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:37.003055 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:37.006183 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:37.006186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:37.006189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:37.006216 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:37.007868 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:37.008466 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:37.008474 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:37.008475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:37.008584 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:37.009020 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:37.009022 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:37.009024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:37.009217 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:37.009315 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:37.010025 (+ 710us) server_negotiation.cc:299] Negotiation successful
0619 12:35:37.010198 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:35:37.011354 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:36.996082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46845 (local address 127.0.98.1:51013)
0619 12:35:36.996591 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:36.996644 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:36.998248 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:36.998830 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:36.998842 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:36.999214 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:36.999823 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:36.999839 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:37.001091 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:37.001096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:37.002518 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:37.002532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:37.003125 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:37.003130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:37.003358 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:37.004213 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:37.004237 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:37.005971 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:37.007998 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:37.008006 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:37.008024 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:37.008313 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:37.008681 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:37.008685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:37.008687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:37.008891 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:37.009344 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:37.009353 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:37.009727 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 12:35:37.009953 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 12:35:37.010358 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":135,"threads_started":1}
I20260619 12:35:37.012763 3650 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46845
I20260619 12:35:37.013031 3650 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:37.013561 3650 heartbeater.cc:511] Master 127.0.98.62:46845 requested a full tablet report, sending...
I20260619 12:35:37.015161 3459 ts_manager.cc:194] Registered new tserver with Master: bd0b9e0ab6ce4d5fb093aa2fbee9d1b5 (127.0.98.1:36797)
I20260619 12:35:37.016388 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:35:37.026378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46845
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:37.139392 3655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:37.139714 3655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:37.139761 3655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:37.143785 3655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:37.143844 3655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:37.143904 3655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:35:37.149263 3655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:37.150377 3655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:37.151397 3655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:37.161369 3663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:37.161381 3660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:37.161448 3655 server_base.cc:1034] running on GCE node
W20260619 12:35:37.161369 3661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:37.162107 3655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:37.162755 3655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:37.163944 3655 hybrid_clock.cc:648] HybridClock initialized: now 1781872537163909 us; error 49 us; skew 500 ppm
Jun 19 12:35:37 dist-test-slave-kdrb krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:37.166671 3655 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:35:37.167776 3655 webserver.cc:460] Webserver started at http://127.0.98.2:42517/ using document root <none> and password file <none>
I20260619 12:35:37.168318 3655 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:37.168378 3655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:37.168546 3655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:37.170607 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "b7742d7646ad4605ac7315763e27792c"
format_stamp: "Formatted at 2026-06-19 12:35:37 on dist-test-slave-kdrb"
server_key: "e4554c7fe6129727343ef4829ff5b7eb"
server_key_iv: "3cbc1615f46c582dff7ff65eeb4d3c79"
server_key_version: "encryptionkey@0"
I20260619 12:35:37.171195 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b7742d7646ad4605ac7315763e27792c"
format_stamp: "Formatted at 2026-06-19 12:35:37 on dist-test-slave-kdrb"
server_key: "e4554c7fe6129727343ef4829ff5b7eb"
server_key_iv: "3cbc1615f46c582dff7ff65eeb4d3c79"
server_key_version: "encryptionkey@0"
I20260619 12:35:37.175462 3655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 12:35:37.178125 3670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:37.179531 3655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:35:37.179652 3655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "b7742d7646ad4605ac7315763e27792c"
format_stamp: "Formatted at 2026-06-19 12:35:37 on dist-test-slave-kdrb"
server_key: "e4554c7fe6129727343ef4829ff5b7eb"
server_key_iv: "3cbc1615f46c582dff7ff65eeb4d3c79"
server_key_version: "encryptionkey@0"
I20260619 12:35:37.179750 3655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:37.198545 3655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:37.201807 3655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:37.201980 3655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:37.202535 3655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:37.203392 3655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:37.203440 3655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:37.203572 3655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:37.203600 3655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:37.214293 3655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41895
I20260619 12:35:37.214319 3783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41895 every 8 connection(s)
I20260619 12:35:37.215245 3655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:35:37.223308 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 3655
I20260619 12:35:37.223445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:35:37.223822 392 external_mini_cluster.cc:1428] Setting key ce7f6655cc38bd0d1e14dea8b5df9dc1
Jun 19 12:35:37 dist-test-slave-kdrb krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:37.229943 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:37.217230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53505 (local address 127.0.98.62:46845)
0619 12:35:37.217394 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:37.217399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:37.218295 (+ 896us) server_negotiation.cc:407] Connection header received
0619 12:35:37.219769 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:37.219773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:37.219840 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:37.219938 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:37.221136 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:37.221653 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:37.222757 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:37.222972 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:37.223086 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:37.226169 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:37.226174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:37.226176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:37.226208 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:37.227778 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:37.228315 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:37.228321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:37.228323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:37.228403 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:37.228773 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:37.228776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:37.228778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:37.228981 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:37.229106 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:37.229646 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 12:35:37.229776 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 12:35:37.230865 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:37.217677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46845 (local address 127.0.98.2:53505)
0619 12:35:37.218137 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:37.218182 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:37.219542 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:37.220111 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:37.220118 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:37.220506 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:37.221009 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:37.221023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:37.221760 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:37.221764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:37.222653 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:37.222663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:37.223119 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:37.223122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:37.223279 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:37.224075 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:37.224095 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:37.225992 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:37.227900 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:37.227907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:37.227924 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:37.228207 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:37.228497 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:37.228501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:37.228503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:37.228657 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:37.229118 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:37.229127 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:37.229399 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 12:35:37.229598 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 12:35:37.229960 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260619 12:35:37.232092 3784 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46845
I20260619 12:35:37.232369 3784 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:37.232904 3784 heartbeater.cc:511] Master 127.0.98.62:46845 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:35:37.234027 3459 ts_manager.cc:194] Registered new tserver with Master: b7742d7646ad4605ac7315763e27792c (127.0.98.2:41895)
I20260619 12:35:37.234673 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:35:37.251217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46845
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:37.364432 3789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:37.364732 3789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:37.364816 3789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:37.369261 3789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:37.369333 3789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:37.369450 3789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:35:37.374869 3789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:37.376022 3789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:37.377123 3789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:37.386770 3797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:37.386801 3794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:37.386780 3795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:37.387123 3789 server_base.cc:1034] running on GCE node
I20260619 12:35:37.387620 3789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:37.388221 3789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:37.389387 3789 hybrid_clock.cc:648] HybridClock initialized: now 1781872537389367 us; error 35 us; skew 500 ppm
Jun 19 12:35:37 dist-test-slave-kdrb krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:37.392230 3789 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:35:37.393183 3789 webserver.cc:460] Webserver started at http://127.0.98.3:34153/ using document root <none> and password file <none>
I20260619 12:35:37.393776 3789 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:37.393839 3789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:37.394065 3789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:37.396224 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "d689756175854d3aab218acc999db033"
format_stamp: "Formatted at 2026-06-19 12:35:37 on dist-test-slave-kdrb"
server_key: "1aafd9ee3fa8b958e84efdb8d7af9038"
server_key_iv: "55c0db46f66b1b649002a5c9ee2cdd41"
server_key_version: "encryptionkey@0"
I20260619 12:35:37.396797 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d689756175854d3aab218acc999db033"
format_stamp: "Formatted at 2026-06-19 12:35:37 on dist-test-slave-kdrb"
server_key: "1aafd9ee3fa8b958e84efdb8d7af9038"
server_key_iv: "55c0db46f66b1b649002a5c9ee2cdd41"
server_key_version: "encryptionkey@0"
I20260619 12:35:37.400605 3789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 12:35:37.403080 3804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:37.404371 3789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:35:37.404526 3789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "d689756175854d3aab218acc999db033"
format_stamp: "Formatted at 2026-06-19 12:35:37 on dist-test-slave-kdrb"
server_key: "1aafd9ee3fa8b958e84efdb8d7af9038"
server_key_iv: "55c0db46f66b1b649002a5c9ee2cdd41"
server_key_version: "encryptionkey@0"
I20260619 12:35:37.404644 3789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:37.414325 3789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:37.417841 3789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:37.418057 3789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:37.418673 3789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:37.419574 3789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:37.419641 3789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:37.419710 3789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:37.419759 3789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:37.428973 3789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38923
I20260619 12:35:37.429034 3917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38923 every 8 connection(s)
I20260619 12:35:37.429952 3789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:35:37.437970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 3789
I20260619 12:35:37.438124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:35:37.438407 392 external_mini_cluster.cc:1428] Setting key 3085f3c415829372c264d792fd85ba12
Jun 19 12:35:37 dist-test-slave-kdrb krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:37.444787 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:37.432044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59279 (local address 127.0.98.62:46845)
0619 12:35:37.432182 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:37.432186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:37.433178 (+ 992us) server_negotiation.cc:407] Connection header received
0619 12:35:37.434686 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:37.434689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:37.434754 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:37.434866 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:37.436232 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:37.436771 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:37.437955 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:37.438231 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:37.438351 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:37.441363 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:37.441366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:37.441368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:37.441394 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:37.442725 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:37.443247 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:37.443249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:37.443251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:37.443325 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:37.443702 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:37.443705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:37.443706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:37.443911 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:37.443992 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:37.444508 (+ 516us) server_negotiation.cc:299] Negotiation successful
0619 12:35:37.444633 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 12:35:37.445580 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:37.432515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46845 (local address 127.0.98.3:59279)
0619 12:35:37.432999 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:37.433045 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:37.434430 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:37.435061 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:37.435069 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:37.435532 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:37.436063 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:37.436077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:37.436905 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:37.436909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:37.437817 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:37.437826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:37.438404 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:37.438408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:37.438623 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:37.439334 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:37.439354 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:37.441210 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:37.442846 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:37.442851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:37.442864 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:37.443153 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:37.443404 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:37.443407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:37.443408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:37.443620 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:37.444004 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:37.444010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:37.444257 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 12:35:37.444411 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 12:35:37.444677 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":137,"threads_started":1}
I20260619 12:35:37.446769 3918 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46845
I20260619 12:35:37.447057 3918 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:37.447678 3918 heartbeater.cc:511] Master 127.0.98.62:46845 requested a full tablet report, sending...
I20260619 12:35:37.448768 3459 ts_manager.cc:194] Registered new tserver with Master: d689756175854d3aab218acc999db033 (127.0.98.3:38923)
I20260619 12:35:37.449359 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59279
I20260619 12:35:37.452466 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 12:35:37 dist-test-slave-kdrb krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872537, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 12:35:37 dist-test-slave-kdrb krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872537, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:37.478060 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:37.469396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33406 (local address 127.0.98.62:46845)
0619 12:35:37.469574 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:37.469579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:37.469705 (+ 126us) server_negotiation.cc:407] Connection header received
0619 12:35:37.469915 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:37.469918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:37.469962 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:37.470050 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:37.470784 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:37.471307 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:37.472218 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:37.472446 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:37.472573 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:37.474904 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:37.474908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:37.474910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:37.474935 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:37.476330 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:37.476759 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:37.476763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:37.476765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:37.476848 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:37.477137 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:37.477141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:37.477142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:37.477372 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:37.477516 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:37.477760 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 12:35:37.477893 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 12:35:37.478513 3459 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33406
I20260619 12:35:37.479586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 3521
I20260619 12:35:37.485580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 3655
I20260619 12:35:37.490821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 3789
I20260619 12:35:37.496196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 3427
2026-06-19T12:35:37Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2434 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:35:37 dist-test-slave-kdrb krb5kdc[3930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:35:37 dist-test-slave-kdrb krb5kdc[3930](info): set up 2 sockets
Jun 19 12:35:37 dist-test-slave-kdrb krb5kdc[3930](info): commencing operation
krb5kdc: starting...
W20260619 12:35:38.535620 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:35:38 dist-test-slave-kdrb krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:35:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:35:38.667883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44737
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:38.780371 3943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:38.780751 3943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:38.780848 3943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:38.785058 3943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:38.785156 3943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:38.785197 3943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:38.785235 3943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:38.785271 3943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:38.790681 3943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:38.791963 3943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:38.793068 3943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:38.802018 3949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:38.802033 3951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:38.802194 3943 server_base.cc:1034] running on GCE node
W20260619 12:35:38.802031 3948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:38.802989 3943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:38.804244 3943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:38.805449 3943 hybrid_clock.cc:648] HybridClock initialized: now 1781872538805435 us; error 32 us; skew 500 ppm
Jun 19 12:35:38 dist-test-slave-kdrb krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:38.808310 3943 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:35:38.809357 3943 webserver.cc:460] Webserver started at http://127.0.98.62:38165/ using document root <none> and password file <none>
I20260619 12:35:38.809950 3943 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:38.810015 3943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:38.810235 3943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:38.812408 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "00a5be7ec777460499cb51c80d27babd"
format_stamp: "Formatted at 2026-06-19 12:35:38 on dist-test-slave-kdrb"
server_key: "9e8d18d0e32d226be3338665ae445951"
server_key_iv: "652595aef4d9bec3c6a9fbc22d2466cd"
server_key_version: "encryptionkey@0"
I20260619 12:35:38.813005 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "00a5be7ec777460499cb51c80d27babd"
format_stamp: "Formatted at 2026-06-19 12:35:38 on dist-test-slave-kdrb"
server_key: "9e8d18d0e32d226be3338665ae445951"
server_key_iv: "652595aef4d9bec3c6a9fbc22d2466cd"
server_key_version: "encryptionkey@0"
I20260619 12:35:38.817178 3943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:35:38.820088 3958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:38.821480 3943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:35:38.821630 3943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "00a5be7ec777460499cb51c80d27babd"
format_stamp: "Formatted at 2026-06-19 12:35:38 on dist-test-slave-kdrb"
server_key: "9e8d18d0e32d226be3338665ae445951"
server_key_iv: "652595aef4d9bec3c6a9fbc22d2466cd"
server_key_version: "encryptionkey@0"
I20260619 12:35:38.821758 3943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:38.839588 3943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:38.843209 3943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:38.843461 3943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:38.851126 3943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44737
I20260619 12:35:38.851156 4010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44737 every 8 connection(s)
I20260619 12:35:38.852169 3943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:38.854904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 3943
I20260619 12:35:38.855064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:38.855189 4011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:38.855412 392 external_mini_cluster.cc:1428] Setting key b4a732fac9070841c919ac4f846e737b
I20260619 12:35:38.862046 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd: Bootstrap starting.
I20260619 12:35:38.864607 4011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd: Neither blocks nor log segments found. Creating new log.
Jun 19 12:35:38 dist-test-slave-kdrb krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:38.865558 4011 log.cc:824] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:38.868203 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd: No bootstrap required, opened a new log
I20260619 12:35:38.870960 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:38.856960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40356 (local address 127.0.98.62:44737)
0619 12:35:38.857453 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:38.857468 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:38.857506 (+ 38us) server_negotiation.cc:407] Connection header received
0619 12:35:38.858183 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:38.858203 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:38.858558 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:38.858947 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:38.859826 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:38.860847 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:38.861727 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:38.862249 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:38.862442 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:38.865445 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:38.865451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:38.865467 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:38.865501 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:38.867665 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:38.868252 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:38.868257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:38.868264 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:38.868464 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:38.868825 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:38.868828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:38.868830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:38.869275 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:38.869460 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:38.869791 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 12:35:38.870172 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":154,"threads_started":1}
I20260619 12:35:38.872658 4011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a5be7ec777460499cb51c80d27babd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44737 } }
I20260619 12:35:38.872941 4011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:38.873026 4011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a5be7ec777460499cb51c80d27babd, State: Initialized, Role: FOLLOWER
I20260619 12:35:38.873543 4011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a5be7ec777460499cb51c80d27babd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44737 } }
I20260619 12:35:38.873739 4011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:38.873821 4011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:38.873924 4011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:38.874935 4011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a5be7ec777460499cb51c80d27babd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44737 } }
I20260619 12:35:38.875309 4011 leader_election.cc:302] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00a5be7ec777460499cb51c80d27babd; no voters:
I20260619 12:35:38.875681 4011 leader_election.cc:288] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:38.875895 4016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:38.876104 4016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [term 1 LEADER]: Becoming Leader. State: Replica: 00a5be7ec777460499cb51c80d27babd, State: Running, Role: LEADER
I20260619 12:35:38.876480 4016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a5be7ec777460499cb51c80d27babd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44737 } }
I20260619 12:35:38.877125 4011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:38.878497 4018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00a5be7ec777460499cb51c80d27babd. Latest consensus state: current_term: 1 leader_uuid: "00a5be7ec777460499cb51c80d27babd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a5be7ec777460499cb51c80d27babd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44737 } } }
I20260619 12:35:38.878703 4018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:38.878497 4017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00a5be7ec777460499cb51c80d27babd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a5be7ec777460499cb51c80d27babd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44737 } } }
I20260619 12:35:38.878809 4017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:38.879393 4025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:38.883352 4025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:38.890259 4025 catalog_manager.cc:1347] Generated new cluster ID: 43863dd5914a42d59e381e4484c3f694
I20260619 12:35:38.890360 4025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:38.900092 4025 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:38.901209 4025 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:38.912393 4025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00a5be7ec777460499cb51c80d27babd: Generated new TSK 0
I20260619 12:35:38.913182 4025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:35:38.946568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44737
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:39.061734 4037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:39.062064 4037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:39.062150 4037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:39.066375 4037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:39.066504 4037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:39.066614 4037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:35:39.072248 4037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:39.073437 4037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:39.074512 4037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:39.084880 4045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:39.084880 4043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:39.085099 4037 server_base.cc:1034] running on GCE node
W20260619 12:35:39.084949 4042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:39.085680 4037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:39.086439 4037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:39.087631 4037 hybrid_clock.cc:648] HybridClock initialized: now 1781872539087613 us; error 36 us; skew 500 ppm
Jun 19 12:35:39 dist-test-slave-kdrb krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:39.090631 4037 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:35:39.091737 4037 webserver.cc:460] Webserver started at http://127.0.98.1:40883/ using document root <none> and password file <none>
I20260619 12:35:39.092350 4037 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:39.092418 4037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:39.092649 4037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:39.094808 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "2c093e3d83dd4bdb958cd63aa7fac032"
format_stamp: "Formatted at 2026-06-19 12:35:39 on dist-test-slave-kdrb"
server_key: "62a616380e51dd8938b414efa93cffe5"
server_key_iv: "614d95046971f0bf6152101673c031df"
server_key_version: "encryptionkey@0"
I20260619 12:35:39.095417 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2c093e3d83dd4bdb958cd63aa7fac032"
format_stamp: "Formatted at 2026-06-19 12:35:39 on dist-test-slave-kdrb"
server_key: "62a616380e51dd8938b414efa93cffe5"
server_key_iv: "614d95046971f0bf6152101673c031df"
server_key_version: "encryptionkey@0"
I20260619 12:35:39.099789 4037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 12:35:39.103888 4052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:39.105849 4037 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 12:35:39.106012 4037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "2c093e3d83dd4bdb958cd63aa7fac032"
format_stamp: "Formatted at 2026-06-19 12:35:39 on dist-test-slave-kdrb"
server_key: "62a616380e51dd8938b414efa93cffe5"
server_key_iv: "614d95046971f0bf6152101673c031df"
server_key_version: "encryptionkey@0"
I20260619 12:35:39.106189 4037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:39.133672 4037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:39.137054 4037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:39.137282 4037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:39.137920 4037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:39.138806 4037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:39.138885 4037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:39.138958 4037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:39.139009 4037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:39.148213 4037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44293
I20260619 12:35:39.148231 4165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44293 every 8 connection(s)
I20260619 12:35:39.149206 4037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:35:39.154074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 4037
I20260619 12:35:39.154218 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:35:39.154507 392 external_mini_cluster.cc:1428] Setting key 488c3c12247bf7a3129e3ec58316d5cf
Jun 19 12:35:39 dist-test-slave-kdrb krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:35:39.165001 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:39.151048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37495 (local address 127.0.98.62:44737)
0619 12:35:39.151184 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:39.151188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:39.152178 (+ 990us) server_negotiation.cc:407] Connection header received
0619 12:35:39.153559 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:39.153562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:39.153612 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:39.153687 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:39.155237 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:39.155906 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:39.157091 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:39.157502 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:39.157600 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:39.160739 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:39.160743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:39.160746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:39.160796 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:39.162562 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:39.163160 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:39.163164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:39.163166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:39.163264 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:39.163719 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:39.163723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:39.163725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:39.163935 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:39.164048 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:39.164624 (+ 576us) server_negotiation.cc:299] Negotiation successful
0619 12:35:39.164768 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:35:39.165915 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:39.151534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44737 (local address 127.0.98.1:37495)
0619 12:35:39.152007 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:39.152053 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:39.153338 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:39.153864 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:39.153871 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:39.154300 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:39.154921 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:39.154940 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:39.156058 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:39.156061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:39.156954 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:39.156963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:39.157682 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:39.157688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:39.157937 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:39.158621 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:39.158638 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:39.160530 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:39.162719 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:39.162725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:39.162744 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:39.163044 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:39.163375 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:39.163378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:39.163380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:39.163617 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:39.164052 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:39.164062 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:39.164363 (+ 301us) client_negotiation.cc:769] Sending connection context
0619 12:35:39.164568 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 12:35:39.164935 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:35:39.167309 4166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44737
I20260619 12:35:39.167631 4166 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:39.168298 4166 heartbeater.cc:511] Master 127.0.98.62:44737 requested a full tablet report, sending...
I20260619 12:35:39.170186 3975 ts_manager.cc:194] Registered new tserver with Master: 2c093e3d83dd4bdb958cd63aa7fac032 (127.0.98.1:44293)
I20260619 12:35:39.171356 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:35:39.187288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44737
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:39.303833 4171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:39.304128 4171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:39.304172 4171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:39.308384 4171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:39.308458 4171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:39.308521 4171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:35:39.314539 4171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:39.315709 4171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:39.316782 4171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:39.327057 4179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:39.327057 4177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:39.327100 4176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:39.327615 4171 server_base.cc:1034] running on GCE node
I20260619 12:35:39.328073 4171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:39.328735 4171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:39.329927 4171 hybrid_clock.cc:648] HybridClock initialized: now 1781872539329887 us; error 55 us; skew 500 ppm
Jun 19 12:35:39 dist-test-slave-kdrb krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:39.333006 4171 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:35:39.334021 4171 webserver.cc:460] Webserver started at http://127.0.98.2:46049/ using document root <none> and password file <none>
I20260619 12:35:39.334618 4171 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:39.334710 4171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:39.334955 4171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:39.337212 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "9bfa831301414bed9fb4daddd6671cdc"
format_stamp: "Formatted at 2026-06-19 12:35:39 on dist-test-slave-kdrb"
server_key: "23e4c49be8233be5658ba8864ba89830"
server_key_iv: "4c67e742a2e46aba36ef0883555529f9"
server_key_version: "encryptionkey@0"
I20260619 12:35:39.337836 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9bfa831301414bed9fb4daddd6671cdc"
format_stamp: "Formatted at 2026-06-19 12:35:39 on dist-test-slave-kdrb"
server_key: "23e4c49be8233be5658ba8864ba89830"
server_key_iv: "4c67e742a2e46aba36ef0883555529f9"
server_key_version: "encryptionkey@0"
I20260619 12:35:39.341930 4171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:35:39.344621 4186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:39.345937 4171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:35:39.346091 4171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "9bfa831301414bed9fb4daddd6671cdc"
format_stamp: "Formatted at 2026-06-19 12:35:39 on dist-test-slave-kdrb"
server_key: "23e4c49be8233be5658ba8864ba89830"
server_key_iv: "4c67e742a2e46aba36ef0883555529f9"
server_key_version: "encryptionkey@0"
I20260619 12:35:39.346217 4171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:39.368268 4171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:39.371906 4171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:39.372121 4171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:39.372756 4171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:39.373631 4171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:39.373710 4171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:39.373808 4171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:39.373860 4171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:39.383235 4171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33607
I20260619 12:35:39.383261 4299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33607 every 8 connection(s)
I20260619 12:35:39.384388 4171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 12:35:39 dist-test-slave-kdrb krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:39.394572 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 4171
I20260619 12:35:39.394701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:35:39.395001 392 external_mini_cluster.cc:1428] Setting key 09ceeeb1c20911cf4fa182ac6182b21a
I20260619 12:35:39.399551 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:39.386343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39527 (local address 127.0.98.62:44737)
0619 12:35:39.386489 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:39.386494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:39.387432 (+ 938us) server_negotiation.cc:407] Connection header received
0619 12:35:39.388959 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:39.388962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:39.389022 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:39.389132 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:39.390381 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:39.390924 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:39.392184 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:39.392419 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:39.392550 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:39.395325 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:39.395329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:39.395332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:39.395362 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:39.397119 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:39.397691 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:39.397697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:39.397702 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:39.397799 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:39.398154 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:39.398160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:39.398165 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:39.398436 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:39.398544 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:39.399254 (+ 710us) server_negotiation.cc:299] Negotiation successful
0619 12:35:39.399369 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 12:35:39.400900 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:39.386769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44737 (local address 127.0.98.2:39527)
0619 12:35:39.387263 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:39.387311 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:39.388707 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:39.389330 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:39.389338 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:39.389714 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:39.390212 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:39.390225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:39.391060 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:39.391063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:39.392053 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:39.392063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:39.392598 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:39.392602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:39.392779 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:39.393366 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:39.393382 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:39.395110 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:39.397263 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:39.397268 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:39.397299 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:39.397581 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:39.397903 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:39.397906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:39.397907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:39.398059 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:39.398578 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:39.398588 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:39.398963 (+ 375us) client_negotiation.cc:769] Sending connection context
0619 12:35:39.399209 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 12:35:39.399673 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":132,"threads_started":1}
I20260619 12:35:39.402335 4300 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44737
I20260619 12:35:39.402612 4300 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:39.403138 4300 heartbeater.cc:511] Master 127.0.98.62:44737 requested a full tablet report, sending...
I20260619 12:35:39.404309 3975 ts_manager.cc:194] Registered new tserver with Master: 9bfa831301414bed9fb4daddd6671cdc (127.0.98.2:33607)
I20260619 12:35:39.404910 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39527
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:35:39.426273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44737
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:39.541042 4305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:39.541327 4305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:39.541370 4305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:39.545706 4305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:39.545783 4305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:39.545845 4305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:35:39.551354 4305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:39.552615 4305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:39.553639 4305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:39.563858 4310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:39.563836 4313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:39.563836 4311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:39.564143 4305 server_base.cc:1034] running on GCE node
I20260619 12:35:39.564700 4305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:39.565371 4305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:39.566550 4305 hybrid_clock.cc:648] HybridClock initialized: now 1781872539566531 us; error 32 us; skew 500 ppm
Jun 19 12:35:39 dist-test-slave-kdrb krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:39.569545 4305 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:35:39.570549 4305 webserver.cc:460] Webserver started at http://127.0.98.3:45767/ using document root <none> and password file <none>
I20260619 12:35:39.571135 4305 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:39.571200 4305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:39.571513 4305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:39.573717 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "436e53f4be604af8b2577a042bfbda52"
format_stamp: "Formatted at 2026-06-19 12:35:39 on dist-test-slave-kdrb"
server_key: "03122cd15677e6ad27f63c1111b037b4"
server_key_iv: "d33c042dfe1d85d2b17f66339cae36c5"
server_key_version: "encryptionkey@0"
I20260619 12:35:39.574299 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "436e53f4be604af8b2577a042bfbda52"
format_stamp: "Formatted at 2026-06-19 12:35:39 on dist-test-slave-kdrb"
server_key: "03122cd15677e6ad27f63c1111b037b4"
server_key_iv: "d33c042dfe1d85d2b17f66339cae36c5"
server_key_version: "encryptionkey@0"
I20260619 12:35:39.578342 4305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:35:39.581066 4320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:39.582420 4305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:35:39.582582 4305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "436e53f4be604af8b2577a042bfbda52"
format_stamp: "Formatted at 2026-06-19 12:35:39 on dist-test-slave-kdrb"
server_key: "03122cd15677e6ad27f63c1111b037b4"
server_key_iv: "d33c042dfe1d85d2b17f66339cae36c5"
server_key_version: "encryptionkey@0"
I20260619 12:35:39.582716 4305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:39.608551 4305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:39.612129 4305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:39.612349 4305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:39.612993 4305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:39.613878 4305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:39.613926 4305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:39.613997 4305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:39.614030 4305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:39.622656 4305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39915
I20260619 12:35:39.622720 4433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39915 every 8 connection(s)
I20260619 12:35:39.623723 4305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:35:39.633461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 4305
I20260619 12:35:39.633579 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:35:39.633883 392 external_mini_cluster.cc:1428] Setting key 293806fb7c5dcc870ddc163b3b9a1d9e
Jun 19 12:35:39 dist-test-slave-kdrb krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:39.638415 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:39.625650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39163 (local address 127.0.98.62:44737)
0619 12:35:39.625844 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:39.625848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:39.626713 (+ 865us) server_negotiation.cc:407] Connection header received
0619 12:35:39.628226 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:39.628230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:39.628289 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:39.628406 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:39.629629 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:39.630167 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:39.631351 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:39.631657 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:39.631772 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:39.634648 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:39.634651 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:39.634653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:39.634681 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:39.636474 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:39.636982 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:39.636984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:39.636985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:39.637060 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:39.637384 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:39.637387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:39.637388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:39.637567 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:39.637644 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:39.638150 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 12:35:39.638257 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:35:39.639374 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:39.626101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44737 (local address 127.0.98.3:39163)
0619 12:35:39.626559 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:39.626604 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:39.627996 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:39.628584 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:39.628592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:39.628989 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:39.629482 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:39.629494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:39.630306 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:39.630309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:39.631218 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:39.631226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:39.631833 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:39.631837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:39.632019 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:39.632694 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:39.632712 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:39.634476 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:39.636581 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:39.636585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:39.636601 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:39.636887 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:39.637139 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:39.637141 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:39.637142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:39.637295 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:39.637654 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:39.637659 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:39.637922 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 12:35:39.638104 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 12:35:39.638378 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":125,"threads_started":1}
I20260619 12:35:39.640609 4434 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44737
I20260619 12:35:39.640977 4434 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:39.641531 4434 heartbeater.cc:511] Master 127.0.98.62:44737 requested a full tablet report, sending...
I20260619 12:35:39.642606 3975 ts_manager.cc:194] Registered new tserver with Master: 436e53f4be604af8b2577a042bfbda52 (127.0.98.3:39915)
I20260619 12:35:39.643217 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39163
I20260619 12:35:39.647997 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 12:35:39 dist-test-slave-kdrb krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 12:35:39 dist-test-slave-kdrb krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:35:39.674398 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:39.664647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34022 (local address 127.0.98.1:44293)
0619 12:35:39.664993 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:39.665007 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:39.665032 (+ 25us) server_negotiation.cc:407] Connection header received
0619 12:35:39.665081 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:39.665086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:39.665281 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:39.665415 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:39.666134 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:39.666916 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:39.667919 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:39.668352 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:39.668455 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:39.671129 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:39.671151 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:39.671165 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:39.671194 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:39.672789 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:39.673216 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:39.673222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:39.673223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:39.673303 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:39.673537 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:39.673540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:39.673541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:39.673789 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:39.673983 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:39.674122 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 12:35:39.674253 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":140,"threads_started":1}
Jun 19 12:35:39 dist-test-slave-kdrb krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:39.686051 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:39.677498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.0.98.62:44737)
0619 12:35:39.677673 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:39.677677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:39.677692 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:35:39.677815 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:39.677818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:39.677859 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:39.677949 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:39.678546 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:39.679286 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:39.680168 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:39.680387 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:39.680495 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:39.682877 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:39.682880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:39.682882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:39.682909 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:39.684266 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:39.684703 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:39.684707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:39.684708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:39.684792 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:39.685117 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:39.685119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:39.685120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:39.685349 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:39.685475 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:39.685690 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 12:35:39.685795 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 12:35:39.686561 3975 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40372
I20260619 12:35:39.687268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 4037
I20260619 12:35:39.693034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 4171
I20260619 12:35:39.698395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 4305
I20260619 12:35:39.703569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 3943
2026-06-19T12:35:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2207 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:35:39 dist-test-slave-kdrb krb5kdc[4446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:35:39 dist-test-slave-kdrb krb5kdc[4446](info): set up 2 sockets
Jun 19 12:35:39 dist-test-slave-kdrb krb5kdc[4446](info): commencing operation
krb5kdc: starting...
W20260619 12:35:40.747851 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:35:40 dist-test-slave-kdrb krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:35:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:35:40.912737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36687
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:41.023241 4459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:41.023638 4459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:41.023749 4459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:41.027818 4459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:41.027916 4459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:41.027956 4459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:41.027992 4459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:41.028026 4459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:41.033437 4459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:41.034543 4459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:41.035634 4459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:41.043991 4465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:41.044117 4459 server_base.cc:1034] running on GCE node
W20260619 12:35:41.043991 4464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:41.043993 4467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:41.044799 4459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:41.045917 4459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:41.047108 4459 hybrid_clock.cc:648] HybridClock initialized: now 1781872541047095 us; error 44 us; skew 500 ppm
Jun 19 12:35:41 dist-test-slave-kdrb krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:41.049851 4459 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:35:41.050633 4459 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:41.050729 4459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:41.050966 4459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:41.053058 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "380b3d32b61541c0b895d058c5e830e5"
format_stamp: "Formatted at 2026-06-19 12:35:41 on dist-test-slave-kdrb"
server_key: "f2e3e30cdc414818c5028f8e5dc5dc9a"
server_key_iv: "2f7fee57f3d44d2df6ac4db48dcee579"
server_key_version: "encryptionkey@0"
I20260619 12:35:41.053622 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "380b3d32b61541c0b895d058c5e830e5"
format_stamp: "Formatted at 2026-06-19 12:35:41 on dist-test-slave-kdrb"
server_key: "f2e3e30cdc414818c5028f8e5dc5dc9a"
server_key_iv: "2f7fee57f3d44d2df6ac4db48dcee579"
server_key_version: "encryptionkey@0"
I20260619 12:35:41.057272 4459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 12:35:41.059579 4473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:41.060760 4459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:35:41.060916 4459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "380b3d32b61541c0b895d058c5e830e5"
format_stamp: "Formatted at 2026-06-19 12:35:41 on dist-test-slave-kdrb"
server_key: "f2e3e30cdc414818c5028f8e5dc5dc9a"
server_key_iv: "2f7fee57f3d44d2df6ac4db48dcee579"
server_key_version: "encryptionkey@0"
I20260619 12:35:41.061030 4459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:41.071771 4459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:41.074527 4459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:41.074761 4459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:41.081599 4459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36687
I20260619 12:35:41.081854 4525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36687 every 8 connection(s)
I20260619 12:35:41.082291 4459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:41.085036 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:41.089107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 4459
I20260619 12:35:41.089262 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:41.089522 392 external_mini_cluster.cc:1428] Setting key d8c9c926f66b6232ef28a5a477eff6b0
I20260619 12:35:41.090843 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5: Bootstrap starting.
I20260619 12:35:41.093261 4526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:41.094008 4526 log.cc:824] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:41.096019 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5: No bootstrap required, opened a new log
Jun 19 12:35:41 dist-test-slave-kdrb krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:41.100598 4526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380b3d32b61541c0b895d058c5e830e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36687 } }
I20260619 12:35:41.100924 4526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:41.100991 4526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380b3d32b61541c0b895d058c5e830e5, State: Initialized, Role: FOLLOWER
I20260619 12:35:41.101514 4526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380b3d32b61541c0b895d058c5e830e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36687 } }
I20260619 12:35:41.101686 4526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:41.101768 4526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:41.101893 4526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:41.103129 4526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380b3d32b61541c0b895d058c5e830e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36687 } }
I20260619 12:35:41.103534 4526 leader_election.cc:302] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 380b3d32b61541c0b895d058c5e830e5; no voters:
I20260619 12:35:41.103896 4526 leader_election.cc:288] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:41.104022 4531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:41.104254 4531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [term 1 LEADER]: Becoming Leader. State: Replica: 380b3d32b61541c0b895d058c5e830e5, State: Running, Role: LEADER
I20260619 12:35:41.104436 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:41.090793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58758 (local address 127.0.98.62:36687)
0619 12:35:41.091244 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:41.091263 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:41.091297 (+ 34us) server_negotiation.cc:407] Connection header received
0619 12:35:41.091981 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:41.091998 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:41.092320 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:41.092620 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:41.093516 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.094536 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.095421 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.096111 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.096275 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:41.098874 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:41.098882 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:41.098904 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:41.098947 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:41.101093 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:41.101532 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:41.101539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:41.101546 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:41.101690 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:41.102073 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:41.102076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:41.102078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:41.102612 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:41.102806 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:41.103029 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 12:35:41.103349 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":148,"threads_started":1}
I20260619 12:35:41.104617 4531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380b3d32b61541c0b895d058c5e830e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36687 } }
I20260619 12:35:41.105217 4526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:41.106320 4533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 380b3d32b61541c0b895d058c5e830e5. Latest consensus state: current_term: 1 leader_uuid: "380b3d32b61541c0b895d058c5e830e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380b3d32b61541c0b895d058c5e830e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36687 } } }
I20260619 12:35:41.106443 4533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:41.106750 4532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "380b3d32b61541c0b895d058c5e830e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380b3d32b61541c0b895d058c5e830e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36687 } } }
I20260619 12:35:41.106824 4532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:41.107225 4540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:41.110435 4540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:41.116641 4540 catalog_manager.cc:1347] Generated new cluster ID: 350ded738e2c45fe834ca8968de53b5e
I20260619 12:35:41.116726 4540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:41.130172 4540 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:41.131453 4540 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:41.138756 4540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5: Generated new TSK 0
I20260619 12:35:41.139578 4540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:35:41.175156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36687
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:41.306298 4552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:41.306571 4552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:41.306650 4552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:41.310725 4552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:41.310798 4552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:41.310863 4552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:35:41.316229 4552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:41.317278 4552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:41.318250 4552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:41.327728 4560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:41.327728 4557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:41.327831 4558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:41.328166 4552 server_base.cc:1034] running on GCE node
I20260619 12:35:41.328730 4552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:41.329414 4552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:41.330607 4552 hybrid_clock.cc:648] HybridClock initialized: now 1781872541330588 us; error 31 us; skew 500 ppm
Jun 19 12:35:41 dist-test-slave-kdrb krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:41.333664 4552 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:35:41.334435 4552 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:41.334506 4552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:41.334685 4552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:41.336864 4552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "6f3fbea4cbf64e61a1dba333fae52b0d"
format_stamp: "Formatted at 2026-06-19 12:35:41 on dist-test-slave-kdrb"
server_key: "129927a943a43852af5b369938e723c6"
server_key_iv: "8429527d6e631f6da1cc00d4509cefbe"
server_key_version: "encryptionkey@0"
I20260619 12:35:41.337393 4552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6f3fbea4cbf64e61a1dba333fae52b0d"
format_stamp: "Formatted at 2026-06-19 12:35:41 on dist-test-slave-kdrb"
server_key: "129927a943a43852af5b369938e723c6"
server_key_iv: "8429527d6e631f6da1cc00d4509cefbe"
server_key_version: "encryptionkey@0"
I20260619 12:35:41.341244 4552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:35:41.343732 4566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:41.345000 4552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 12:35:41.345140 4552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "6f3fbea4cbf64e61a1dba333fae52b0d"
format_stamp: "Formatted at 2026-06-19 12:35:41 on dist-test-slave-kdrb"
server_key: "129927a943a43852af5b369938e723c6"
server_key_iv: "8429527d6e631f6da1cc00d4509cefbe"
server_key_version: "encryptionkey@0"
I20260619 12:35:41.345256 4552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:41.370585 4552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:41.373915 4552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:41.374157 4552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:41.374799 4552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:41.375752 4552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:41.375826 4552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:41.375905 4552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:41.375947 4552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:41.385289 4552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38749
I20260619 12:35:41.385305 4679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38749 every 8 connection(s)
I20260619 12:35:41.385924 4552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:35:41.392781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 4552
I20260619 12:35:41.392931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:35:41.393208 392 external_mini_cluster.cc:1428] Setting key 38b30d83698e127885711cb312cd09ec
Jun 19 12:35:41 dist-test-slave-kdrb krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:41.400818 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:41.387785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46361 (local address 127.0.98.62:36687)
0619 12:35:41.387926 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:41.387930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:41.388729 (+ 799us) server_negotiation.cc:407] Connection header received
0619 12:35:41.390154 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:41.390158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:41.390213 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:41.390290 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:41.391523 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.392072 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.393439 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.393758 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.393895 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:41.396808 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:41.396812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:41.396814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:41.396838 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:41.398285 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:41.398998 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:41.399001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:41.399003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:41.399098 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:41.399574 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:41.399577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:41.399581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:41.399772 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:41.399854 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:41.400585 (+ 731us) server_negotiation.cc:299] Negotiation successful
0619 12:35:41.400695 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:35:41.402168 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:41.388127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36687 (local address 127.0.98.1:46361)
0619 12:35:41.388578 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:41.388623 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:41.389934 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:41.390481 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:41.390489 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:41.390853 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:41.391344 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:41.391357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.392216 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.392220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:41.393299 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:41.393312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.393928 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.393933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:41.394117 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:41.394764 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:41.394785 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:41.396616 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:41.398411 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:41.398420 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:41.398441 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:41.398853 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:41.399211 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:41.399215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:41.399219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:41.399439 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:41.399893 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:41.399904 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:41.400300 (+ 396us) client_negotiation.cc:769] Sending connection context
0619 12:35:41.400563 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 12:35:41.400978 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:35:41.403554 4680 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36687
I20260619 12:35:41.403803 4680 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:41.404284 4680 heartbeater.cc:511] Master 127.0.98.62:36687 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:35:41.405896 4490 ts_manager.cc:194] Registered new tserver with Master: 6f3fbea4cbf64e61a1dba333fae52b0d (127.0.98.1:38749)
I20260619 12:35:41.407600 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:35:41.423103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36687
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:41.535249 4685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:41.535616 4685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:41.535698 4685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:41.539886 4685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:41.539999 4685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:41.540096 4685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:35:41.545742 4685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:41.546900 4685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:41.548051 4685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:41.558048 4693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:41.558073 4691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:41.558051 4690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:41.558266 4685 server_base.cc:1034] running on GCE node
I20260619 12:35:41.558837 4685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:41.559561 4685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:41.560765 4685 hybrid_clock.cc:648] HybridClock initialized: now 1781872541560731 us; error 52 us; skew 500 ppm
Jun 19 12:35:41 dist-test-slave-kdrb krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:41.563728 4685 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:35:41.564563 4685 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:41.564662 4685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:41.564900 4685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:41.566975 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "65b3b0fe3c274199a28be683b3938288"
format_stamp: "Formatted at 2026-06-19 12:35:41 on dist-test-slave-kdrb"
server_key: "5145fb24a4a31489ce652ed9f27b801c"
server_key_iv: "e2a7257cd19b03c1b021e56ab4deacaa"
server_key_version: "encryptionkey@0"
I20260619 12:35:41.567610 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "65b3b0fe3c274199a28be683b3938288"
format_stamp: "Formatted at 2026-06-19 12:35:41 on dist-test-slave-kdrb"
server_key: "5145fb24a4a31489ce652ed9f27b801c"
server_key_iv: "e2a7257cd19b03c1b021e56ab4deacaa"
server_key_version: "encryptionkey@0"
I20260619 12:35:41.571339 4685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 12:35:41.573863 4699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:41.575102 4685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 12:35:41.575260 4685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "65b3b0fe3c274199a28be683b3938288"
format_stamp: "Formatted at 2026-06-19 12:35:41 on dist-test-slave-kdrb"
server_key: "5145fb24a4a31489ce652ed9f27b801c"
server_key_iv: "e2a7257cd19b03c1b021e56ab4deacaa"
server_key_version: "encryptionkey@0"
I20260619 12:35:41.575380 4685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:41.607833 4685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:41.611313 4685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:41.611580 4685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:41.612190 4685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:41.613068 4685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:41.613142 4685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:41.613214 4685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:41.613263 4685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:41.622215 4685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35293
I20260619 12:35:41.622231 4812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35293 every 8 connection(s)
I20260619 12:35:41.622871 4685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:35:41.630286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 4685
I20260619 12:35:41.630419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:35:41.630695 392 external_mini_cluster.cc:1428] Setting key 7b6fd10e8e893ea3e44f04f3d851aa36
Jun 19 12:35:41 dist-test-slave-kdrb krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:41.637727 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:41.624725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51541 (local address 127.0.98.62:36687)
0619 12:35:41.624875 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:41.624879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:41.625791 (+ 912us) server_negotiation.cc:407] Connection header received
0619 12:35:41.627186 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:41.627190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:41.627252 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:41.627356 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:41.628619 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.629152 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.630491 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.630766 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.630902 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:41.633948 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:41.633952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:41.633954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:41.633988 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:41.635581 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:41.636141 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:41.636146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:41.636148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:41.636232 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:41.636588 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:41.636595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:41.636596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:41.636795 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:41.636914 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:41.637433 (+ 519us) server_negotiation.cc:299] Negotiation successful
0619 12:35:41.637559 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:35:41.638670 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:41.625158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36687 (local address 127.0.98.2:51541)
0619 12:35:41.625628 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:41.625672 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:41.626966 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:41.627599 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:41.627607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:41.627979 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:41.628469 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:41.628482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.629276 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.629285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:41.630313 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:41.630325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.630957 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.630961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:41.631173 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:41.631937 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:41.631959 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:41.633735 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:41.635706 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:41.635713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:41.635730 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:41.636031 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:41.636336 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:41.636339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:41.636341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:41.636491 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:41.636902 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:41.636909 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:41.637183 (+ 274us) client_negotiation.cc:769] Sending connection context
0619 12:35:41.637386 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 12:35:41.637773 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:35:41.640033 4813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36687
I20260619 12:35:41.640313 4813 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:41.640842 4813 heartbeater.cc:511] Master 127.0.98.62:36687 requested a full tablet report, sending...
I20260619 12:35:41.641986 4490 ts_manager.cc:194] Registered new tserver with Master: 65b3b0fe3c274199a28be683b3938288 (127.0.98.2:35293)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:35:41.642593 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:35:41.657864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36687
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:41.769114 4818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:41.769412 4818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:41.769506 4818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:41.773629 4818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:41.773703 4818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:41.773805 4818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:35:41.779871 4818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:41.781031 4818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:41.782146 4818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:41.791811 4823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:41.791882 4818 server_base.cc:1034] running on GCE node
W20260619 12:35:41.791831 4824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:41.791821 4826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:41.792526 4818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:41.793236 4818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:41.794428 4818 hybrid_clock.cc:648] HybridClock initialized: now 1781872541794402 us; error 43 us; skew 500 ppm
Jun 19 12:35:41 dist-test-slave-kdrb krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:41.807412 4818 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:35:41.808274 4818 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:41.808342 4818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:41.808529 4818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:41.810616 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "8be943d5f6e24d0cb1f315af0f9ed590"
format_stamp: "Formatted at 2026-06-19 12:35:41 on dist-test-slave-kdrb"
server_key: "f26fa482b4e3e91b85c74dff28fd3570"
server_key_iv: "7733a8ebaad911674f310809c7a1fcec"
server_key_version: "encryptionkey@0"
I20260619 12:35:41.811153 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8be943d5f6e24d0cb1f315af0f9ed590"
format_stamp: "Formatted at 2026-06-19 12:35:41 on dist-test-slave-kdrb"
server_key: "f26fa482b4e3e91b85c74dff28fd3570"
server_key_iv: "7733a8ebaad911674f310809c7a1fcec"
server_key_version: "encryptionkey@0"
I20260619 12:35:41.815004 4818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:35:41.817641 4832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:41.818926 4818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:35:41.819059 4818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "8be943d5f6e24d0cb1f315af0f9ed590"
format_stamp: "Formatted at 2026-06-19 12:35:41 on dist-test-slave-kdrb"
server_key: "f26fa482b4e3e91b85c74dff28fd3570"
server_key_iv: "7733a8ebaad911674f310809c7a1fcec"
server_key_version: "encryptionkey@0"
I20260619 12:35:41.819178 4818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:41.837785 4818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:41.841362 4818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:41.841598 4818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:41.842223 4818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:41.843091 4818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:41.843142 4818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:41.843211 4818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:41.843236 4818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:41.852423 4818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37873
I20260619 12:35:41.852450 4945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37873 every 8 connection(s)
I20260619 12:35:41.853056 4818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:35:41.854741 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 4818
I20260619 12:35:41.854880 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:35:41.855188 392 external_mini_cluster.cc:1428] Setting key d8458ea89ec9c331afed67d502d71f5a
Jun 19 12:35:41 dist-test-slave-kdrb krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:41.867362 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:41.854823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41327 (local address 127.0.98.62:36687)
0619 12:35:41.855001 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:41.855007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:41.856031 (+ 1024us) server_negotiation.cc:407] Connection header received
0619 12:35:41.857864 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:41.857866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:41.857913 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:41.857990 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:41.859173 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.859814 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.861002 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.861232 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.861335 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:41.863983 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:41.863986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:41.863988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:41.864013 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:41.865525 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:41.866023 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:41.866026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:41.866027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:41.866099 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:41.866401 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:41.866403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:41.866404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:41.866572 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:41.866646 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:41.867141 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 12:35:41.867241 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 12:35:41.868391 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:41.855305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36687 (local address 127.0.98.3:41327)
0619 12:35:41.855867 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:41.855938 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:41.857668 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:41.858130 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:41.858137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:41.858511 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:41.859013 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:41.859026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.859940 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.859944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:41.860862 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:41.860873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.861400 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.861403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:41.861560 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:41.862163 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:41.862193 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:41.863804 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:41.865645 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:41.865649 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:41.865662 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:41.865935 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:41.866182 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:41.866184 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:41.866185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:41.866322 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:41.866674 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:41.866679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:41.866937 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 12:35:41.867100 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 12:35:41.867372 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":158,"threads_started":1}
I20260619 12:35:41.869634 4946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36687
I20260619 12:35:41.869845 4946 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:41.870254 4946 heartbeater.cc:511] Master 127.0.98.62:36687 requested a full tablet report, sending...
I20260619 12:35:41.871203 4489 ts_manager.cc:194] Registered new tserver with Master: 8be943d5f6e24d0cb1f315af0f9ed590 (127.0.98.3:37873)
I20260619 12:35:41.871876 4489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41327
I20260619 12:35:41.881498 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:35:41.891309 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:41.883457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58768 (local address 127.0.98.62:36687)
0619 12:35:41.883624 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:41.883629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:41.883832 (+ 203us) server_negotiation.cc:407] Connection header received
0619 12:35:41.884005 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:41.884008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:41.884061 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:41.884146 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:41.884851 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.885366 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.886220 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.886448 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.886559 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:41.888049 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:41.888053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:41.888056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:41.888091 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:41.889583 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:41.890029 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:41.890034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:41.890036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:41.890119 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:41.890398 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:41.890401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:41.890403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:41.890632 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:41.890774 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:41.891020 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 12:35:41.891140 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 12:35:41.894544 4489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:35:41.897662 4489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:41.912415 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:41.906548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48578 (local address 127.0.98.1:38749)
0619 12:35:41.907379 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:41.907388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:41.907414 (+ 26us) server_negotiation.cc:407] Connection header received
0619 12:35:41.907516 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:41.907521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:41.907751 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:41.907928 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:41.908250 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.909244 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.910843 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.911751 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.911886 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:41.912016 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:41.912180 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 12:35:41.912261 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":67,"threads_started":1}
I20260619 12:35:41.912465 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:41.906436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38749 (local address 127.0.0.1:48578)
0619 12:35:41.906872 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:41.906930 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:41.907057 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:41.907928 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:41.907935 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:41.907963 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:41.908108 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:41.908116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.909377 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.909381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:41.910705 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:41.910714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.911876 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.911879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:41.911989 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:41.912029 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 12:35:41.912149 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 12:35:41.912271 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260619 12:35:41.913033 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:41.907160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35293 (local address 127.0.0.1:44700)
0619 12:35:41.908028 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:41.908043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:41.908123 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:41.908706 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:41.908709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:41.908726 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:41.908828 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:41.908833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.910212 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.910215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:41.911424 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:41.911432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.912464 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.912468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:41.912724 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:41.912744 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 12:35:41.912839 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 12:35:41.912907 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":45,"thread_start_us":104,"threads_started":1}
I20260619 12:35:41.913586 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:41.907625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.0.98.2:35293)
0619 12:35:41.908233 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:41.908243 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:41.908285 (+ 42us) server_negotiation.cc:407] Connection header received
0619 12:35:41.908361 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:41.908365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:41.908564 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:41.908731 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:41.908957 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.910070 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.911592 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.912325 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.912431 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:41.912581 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:41.913209 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 12:35:41.913326 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":110,"threads_started":1}
I20260619 12:35:41.915874 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:41.906508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37873 (local address 127.0.0.1:41474)
0619 12:35:41.907631 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:41.907645 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:41.907739 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:41.909356 (+ 1617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:41.909359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:41.909375 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:41.909470 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:41.909476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.910500 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.910503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:41.911750 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:41.911760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.915379 (+ 3619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.915382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:41.915514 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:41.915609 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 12:35:41.915675 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 12:35:41.915746 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":36,"thread_start_us":90,"threads_started":1}
I20260619 12:35:41.916496 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:41.907614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.0.98.3:37873)
0619 12:35:41.909034 (+ 1420us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:41.909042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:41.909058 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:35:41.909114 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:41.909117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:41.909235 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:41.909378 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:41.909561 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.910369 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.912873 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:41.914952 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:41.916043 (+ 1091us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:41.916199 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:41.916251 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 12:35:41.916345 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1324,"thread_start_us":169,"threads_started":1}
I20260619 12:35:41.916790 4614 tablet_service.cc:1467] Processing CreateTablet for tablet 295c4c3e51324a548fdeb0d51ef25cc9 (DEFAULT_TABLE table=test-table [id=ba8fca6ee2b2414fa8e5df067350c4df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:41.916793 4747 tablet_service.cc:1467] Processing CreateTablet for tablet 295c4c3e51324a548fdeb0d51ef25cc9 (DEFAULT_TABLE table=test-table [id=ba8fca6ee2b2414fa8e5df067350c4df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:41.917835 4747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 295c4c3e51324a548fdeb0d51ef25cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:41.917846 4614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 295c4c3e51324a548fdeb0d51ef25cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:41.919505 4880 tablet_service.cc:1467] Processing CreateTablet for tablet 295c4c3e51324a548fdeb0d51ef25cc9 (DEFAULT_TABLE table=test-table [id=ba8fca6ee2b2414fa8e5df067350c4df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:41.920545 4880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 295c4c3e51324a548fdeb0d51ef25cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:41.924067 4963 tablet_bootstrap.cc:492] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288: Bootstrap starting.
I20260619 12:35:41.926247 4963 tablet_bootstrap.cc:654] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:41.927412 4963 log.cc:824] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:41.927780 4965 tablet_bootstrap.cc:492] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590: Bootstrap starting.
I20260619 12:35:41.928215 4964 tablet_bootstrap.cc:492] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d: Bootstrap starting.
I20260619 12:35:41.929697 4965 tablet_bootstrap.cc:654] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:41.929972 4964 tablet_bootstrap.cc:654] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:41.930820 4965 log.cc:824] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:41.930811 4964 log.cc:824] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:41.931795 4963 tablet_bootstrap.cc:492] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288: No bootstrap required, opened a new log
I20260619 12:35:41.932014 4963 ts_tablet_manager.cc:1397] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 12:35:41.932881 4965 tablet_bootstrap.cc:492] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590: No bootstrap required, opened a new log
I20260619 12:35:41.932986 4964 tablet_bootstrap.cc:492] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d: No bootstrap required, opened a new log
I20260619 12:35:41.933068 4965 ts_tablet_manager.cc:1397] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:35:41.933193 4964 ts_tablet_manager.cc:1397] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 12:35:41.936112 4963 raft_consensus.cc:348] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3fbea4cbf64e61a1dba333fae52b0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38749 } } peers { permanent_uuid: "8be943d5f6e24d0cb1f315af0f9ed590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37873 } } peers { permanent_uuid: "65b3b0fe3c274199a28be683b3938288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35293 } }
I20260619 12:35:41.936443 4963 raft_consensus.cc:374] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:41.936506 4963 raft_consensus.cc:729] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b3b0fe3c274199a28be683b3938288, State: Initialized, Role: FOLLOWER
I20260619 12:35:41.936738 4964 raft_consensus.cc:348] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3fbea4cbf64e61a1dba333fae52b0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38749 } } peers { permanent_uuid: "8be943d5f6e24d0cb1f315af0f9ed590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37873 } } peers { permanent_uuid: "65b3b0fe3c274199a28be683b3938288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35293 } }
I20260619 12:35:41.937011 4964 raft_consensus.cc:374] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:41.937067 4964 raft_consensus.cc:729] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3fbea4cbf64e61a1dba333fae52b0d, State: Initialized, Role: FOLLOWER
I20260619 12:35:41.936882 4965 raft_consensus.cc:348] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3fbea4cbf64e61a1dba333fae52b0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38749 } } peers { permanent_uuid: "8be943d5f6e24d0cb1f315af0f9ed590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37873 } } peers { permanent_uuid: "65b3b0fe3c274199a28be683b3938288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35293 } }
I20260619 12:35:41.937008 4963 consensus_queue.cc:260] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3fbea4cbf64e61a1dba333fae52b0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38749 } } peers { permanent_uuid: "8be943d5f6e24d0cb1f315af0f9ed590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37873 } } peers { permanent_uuid: "65b3b0fe3c274199a28be683b3938288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35293 } }
I20260619 12:35:41.937135 4965 raft_consensus.cc:374] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:41.937189 4965 raft_consensus.cc:729] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be943d5f6e24d0cb1f315af0f9ed590, State: Initialized, Role: FOLLOWER
I20260619 12:35:41.937634 4965 consensus_queue.cc:260] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3fbea4cbf64e61a1dba333fae52b0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38749 } } peers { permanent_uuid: "8be943d5f6e24d0cb1f315af0f9ed590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37873 } } peers { permanent_uuid: "65b3b0fe3c274199a28be683b3938288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35293 } }
I20260619 12:35:41.937634 4964 consensus_queue.cc:260] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3fbea4cbf64e61a1dba333fae52b0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38749 } } peers { permanent_uuid: "8be943d5f6e24d0cb1f315af0f9ed590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37873 } } peers { permanent_uuid: "65b3b0fe3c274199a28be683b3938288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35293 } }
I20260619 12:35:41.937731 4813 heartbeater.cc:503] Master 127.0.98.62:36687 was elected leader, sending a full tablet report...
I20260619 12:35:41.938114 4963 ts_tablet_manager.cc:1428] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260619 12:35:41.938508 4965 ts_tablet_manager.cc:1428] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 12:35:41.938561 4964 ts_tablet_manager.cc:1428] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 12:35:41.938865 4680 heartbeater.cc:503] Master 127.0.98.62:36687 was elected leader, sending a full tablet report...
I20260619 12:35:41.938901 4946 heartbeater.cc:503] Master 127.0.98.62:36687 was elected leader, sending a full tablet report...
I20260619 12:35:42.003994 4970 raft_consensus.cc:482] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:42.004199 4970 raft_consensus.cc:504] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3fbea4cbf64e61a1dba333fae52b0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38749 } } peers { permanent_uuid: "8be943d5f6e24d0cb1f315af0f9ed590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37873 } } peers { permanent_uuid: "65b3b0fe3c274199a28be683b3938288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35293 } }
I20260619 12:35:42.005463 4970 leader_election.cc:288] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f3fbea4cbf64e61a1dba333fae52b0d (127.0.98.1:38749), 65b3b0fe3c274199a28be683b3938288 (127.0.98.2:35293)
I20260619 12:35:42.010717 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.005703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38749 (local address 127.0.98.3:41793)
0619 12:35:42.005853 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:42.005871 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:42.005970 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:42.006435 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:42.006441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:42.006458 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:42.006636 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:42.006644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:42.007703 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:42.007706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:42.009324 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:42.009336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:42.010253 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:42.010256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:42.010359 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:42.010385 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 12:35:42.010472 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 12:35:42.010544 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":267}
I20260619 12:35:42.010933 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.005724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35293 (local address 127.0.98.3:56699)
0619 12:35:42.006105 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:42.006139 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:42.006281 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:42.006527 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:42.006530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:42.006541 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:42.006651 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:42.006656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:42.007703 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:42.007706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:42.009324 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:42.009336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:42.010398 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:42.010424 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:42.010535 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:42.010570 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 12:35:42.010709 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 12:35:42.010785 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":103,"thread_start_us":130,"threads_started":1}
I20260619 12:35:42.010942 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.005961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56699 (local address 127.0.98.2:35293)
0619 12:35:42.006088 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:42.006093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:42.006254 (+ 161us) server_negotiation.cc:407] Connection header received
0619 12:35:42.006388 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:42.006391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:42.006440 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:42.006512 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:42.006763 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:42.007589 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:42.009484 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:42.010110 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:42.010228 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:42.010295 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:42.010733 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 12:35:42.010810 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 12:35:42.011246 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.005835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41793 (local address 127.0.98.1:38749)
0619 12:35:42.006085 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:42.006097 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:42.006111 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:35:42.006206 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:42.006209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:42.006289 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:42.006367 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:42.006764 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:42.007584 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:42.009484 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:42.010127 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:42.010228 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:42.010295 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:42.011017 (+ 722us) server_negotiation.cc:299] Negotiation successful
0619 12:35:42.011127 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260619 12:35:42.011710 4767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "295c4c3e51324a548fdeb0d51ef25cc9" candidate_uuid: "8be943d5f6e24d0cb1f315af0f9ed590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b3b0fe3c274199a28be683b3938288" is_pre_election: true
I20260619 12:35:42.011770 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "295c4c3e51324a548fdeb0d51ef25cc9" candidate_uuid: "8be943d5f6e24d0cb1f315af0f9ed590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3fbea4cbf64e61a1dba333fae52b0d" is_pre_election: true
I20260619 12:35:42.012048 4767 raft_consensus.cc:2454] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8be943d5f6e24d0cb1f315af0f9ed590 in term 0.
I20260619 12:35:42.012048 4634 raft_consensus.cc:2454] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8be943d5f6e24d0cb1f315af0f9ed590 in term 0.
I20260619 12:35:42.012518 4835 leader_election.cc:302] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f3fbea4cbf64e61a1dba333fae52b0d, 8be943d5f6e24d0cb1f315af0f9ed590; no voters:
I20260619 12:35:42.012815 4970 raft_consensus.cc:2781] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:42.012909 4970 raft_consensus.cc:482] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:42.012961 4970 raft_consensus.cc:3037] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:42.014031 4970 raft_consensus.cc:504] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3fbea4cbf64e61a1dba333fae52b0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38749 } } peers { permanent_uuid: "8be943d5f6e24d0cb1f315af0f9ed590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37873 } } peers { permanent_uuid: "65b3b0fe3c274199a28be683b3938288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35293 } }
I20260619 12:35:42.014446 4970 leader_election.cc:288] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [CANDIDATE]: Term 1 election: Requested vote from peers 6f3fbea4cbf64e61a1dba333fae52b0d (127.0.98.1:38749), 65b3b0fe3c274199a28be683b3938288 (127.0.98.2:35293)
I20260619 12:35:42.014880 4767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "295c4c3e51324a548fdeb0d51ef25cc9" candidate_uuid: "8be943d5f6e24d0cb1f315af0f9ed590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b3b0fe3c274199a28be683b3938288"
I20260619 12:35:42.014948 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "295c4c3e51324a548fdeb0d51ef25cc9" candidate_uuid: "8be943d5f6e24d0cb1f315af0f9ed590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3fbea4cbf64e61a1dba333fae52b0d"
I20260619 12:35:42.015012 4767 raft_consensus.cc:3037] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:42.015108 4634 raft_consensus.cc:3037] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:42.016356 4634 raft_consensus.cc:2454] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8be943d5f6e24d0cb1f315af0f9ed590 in term 1.
I20260619 12:35:42.016366 4767 raft_consensus.cc:2454] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8be943d5f6e24d0cb1f315af0f9ed590 in term 1.
I20260619 12:35:42.016724 4835 leader_election.cc:302] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f3fbea4cbf64e61a1dba333fae52b0d, 8be943d5f6e24d0cb1f315af0f9ed590; no voters:
I20260619 12:35:42.016957 4970 raft_consensus.cc:2781] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:42.017141 4970 raft_consensus.cc:686] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [term 1 LEADER]: Becoming Leader. State: Replica: 8be943d5f6e24d0cb1f315af0f9ed590, State: Running, Role: LEADER
I20260619 12:35:42.017495 4970 consensus_queue.cc:237] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3fbea4cbf64e61a1dba333fae52b0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38749 } } peers { permanent_uuid: "8be943d5f6e24d0cb1f315af0f9ed590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37873 } } peers { permanent_uuid: "65b3b0fe3c274199a28be683b3938288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35293 } }
I20260619 12:35:42.020728 4489 catalog_manager.cc:5495] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8be943d5f6e24d0cb1f315af0f9ed590 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8be943d5f6e24d0cb1f315af0f9ed590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3fbea4cbf64e61a1dba333fae52b0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8be943d5f6e24d0cb1f315af0f9ed590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b3b0fe3c274199a28be683b3938288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35293 } health_report { overall_health: UNKNOWN } } }
I20260619 12:35:42.044245 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.040498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.0.98.3:37873)
0619 12:35:42.040767 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:42.040773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:42.040792 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:35:42.040891 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:42.040894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:42.040950 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:42.041069 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:42.041325 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:42.041838 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:42.042696 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:42.042922 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:42.043037 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:42.043346 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:42.043806 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:42.043939 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:42.044015 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 12:35:42.044073 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260619 12:35:42.050185 4634 raft_consensus.cc:1261] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d [term 1 FOLLOWER]: Refusing update from remote peer 8be943d5f6e24d0cb1f315af0f9ed590: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:42.050185 4767 raft_consensus.cc:1261] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288 [term 1 FOLLOWER]: Refusing update from remote peer 8be943d5f6e24d0cb1f315af0f9ed590: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:42.050942 4970 consensus_queue.cc:1035] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f3fbea4cbf64e61a1dba333fae52b0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:42.051151 4973 consensus_queue.cc:1035] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b3b0fe3c274199a28be683b3938288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:42.058952 4978 mvcc.cc:205] Tried to move back new op lower bound from 7298549932230156288 to 7298549932106457088. Current Snapshot: MvccSnapshot[applied={T|T < 7298549932230156288}]
I20260619 12:35:42.068354 4489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:42.068526 4489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:42.071091 4489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 380b3d32b61541c0b895d058c5e830e5: Sending DeleteTablet for 3 replicas of tablet 295c4c3e51324a548fdeb0d51ef25cc9
I20260619 12:35:42.071911 4614 tablet_service.cc:1514] Processing DeleteTablet for tablet 295c4c3e51324a548fdeb0d51ef25cc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48578
I20260619 12:35:42.072168 4880 tablet_service.cc:1514] Processing DeleteTablet for tablet 295c4c3e51324a548fdeb0d51ef25cc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41474
I20260619 12:35:42.072317 4747 tablet_service.cc:1514] Processing DeleteTablet for tablet 295c4c3e51324a548fdeb0d51ef25cc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44700
I20260619 12:35:42.072484 4985 tablet_replica.cc:330] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d: stopping tablet replica
I20260619 12:35:42.072813 4987 tablet_replica.cc:330] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288: stopping tablet replica
I20260619 12:35:42.072804 4985 raft_consensus.cc:2229] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:42.073096 4987 raft_consensus.cc:2229] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:42.073139 4985 raft_consensus.cc:2258] T 295c4c3e51324a548fdeb0d51ef25cc9 P 6f3fbea4cbf64e61a1dba333fae52b0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:42.073390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 4552
I20260619 12:35:42.074900 4986 tablet_replica.cc:330] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590: stopping tablet replica
I20260619 12:35:42.075224 4986 raft_consensus.cc:2229] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:35:42.075690 4986 raft_consensus.cc:2258] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:42.076221 4987 raft_consensus.cc:2258] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:42.077276 4986 ts_tablet_manager.cc:1905] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:42.077976 4987 ts_tablet_manager.cc:1905] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:42.080428 4986 ts_tablet_manager.cc:1918] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:42.080536 4986 log.cc:1196] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/295c4c3e51324a548fdeb0d51ef25cc9
I20260619 12:35:42.080760 4987 ts_tablet_manager.cc:1918] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:42.080864 4987 log.cc:1196] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/295c4c3e51324a548fdeb0d51ef25cc9
I20260619 12:35:42.080880 4986 ts_tablet_manager.cc:1939] T 295c4c3e51324a548fdeb0d51ef25cc9 P 8be943d5f6e24d0cb1f315af0f9ed590: Deleting consensus metadata
I20260619 12:35:42.081175 4987 ts_tablet_manager.cc:1939] T 295c4c3e51324a548fdeb0d51ef25cc9 P 65b3b0fe3c274199a28be683b3938288: Deleting consensus metadata
W20260619 12:35:42.081234 4476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38749 (error 108)
I20260619 12:35:42.082122 4474 catalog_manager.cc:4841] TS 8be943d5f6e24d0cb1f315af0f9ed590 (127.0.98.3:37873): tablet 295c4c3e51324a548fdeb0d51ef25cc9 (table test-table [id=ba8fca6ee2b2414fa8e5df067350c4df]) successfully deleted
I20260619 12:35:42.082417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 4685
I20260619 12:35:42.085008 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.084717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38749 (local address 127.0.0.1:48580)
0619 12:35:42.084857 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:42.084931 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 12:35:42.085429 4476 catalog_manager.cc:4584] TS 6f3fbea4cbf64e61a1dba333fae52b0d (127.0.98.1:38749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 295c4c3e51324a548fdeb0d51ef25cc9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38749: connect: Connection refused (error 111)
I20260619 12:35:42.091347 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 4818
I20260619 12:35:42.092813 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.091523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35293 (local address 127.0.0.1:44714)
0619 12:35:42.092662 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:42.092727 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1047}
I20260619 12:35:42.097275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 4459
2026-06-19T12:35:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2394 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T12:35:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:42Z Disabled control of system clock
I20260619 12:35:42.128613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33715
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 12:35:42.240875 4992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:42.241186 4992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:42.241226 4992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:42.245453 4992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:42.245543 4992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:42.245558 4992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:42.245576 4992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:42.245592 4992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:42.251202 4992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:42.252422 4992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:42.253803 4992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:35:42.262539 4992 server_base.cc:1034] running on GCE node
W20260619 12:35:42.262499 4998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:42.262504 4997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:42.262499 5000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:42.263393 4992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:42.264570 4992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:42.265729 4992 hybrid_clock.cc:648] HybridClock initialized: now 1781872542265696 us; error 51 us; skew 500 ppm
I20260619 12:35:42.267617 4992 webserver.cc:460] Webserver started at http://127.0.98.62:39637/ using document root <none> and password file <none>
I20260619 12:35:42.268198 4992 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:42.268260 4992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:42.268507 4992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:42.270705 4992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "c8161bb05abe4df9950059a6c39b60e9"
format_stamp: "Formatted at 2026-06-19 12:35:42 on dist-test-slave-kdrb"
server_key: "64f34750e84fe8327d1ee4f68f01c22d"
server_key_iv: "6446c0563b90e4e9e3d8db1f9e0ce859"
server_key_version: "encryptionkey@0"
I20260619 12:35:42.271291 4992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "c8161bb05abe4df9950059a6c39b60e9"
format_stamp: "Formatted at 2026-06-19 12:35:42 on dist-test-slave-kdrb"
server_key: "64f34750e84fe8327d1ee4f68f01c22d"
server_key_iv: "6446c0563b90e4e9e3d8db1f9e0ce859"
server_key_version: "encryptionkey@0"
I20260619 12:35:42.274978 4992 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 12:35:42.277423 5006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:42.278612 4992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:35:42.278754 4992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "c8161bb05abe4df9950059a6c39b60e9"
format_stamp: "Formatted at 2026-06-19 12:35:42 on dist-test-slave-kdrb"
server_key: "64f34750e84fe8327d1ee4f68f01c22d"
server_key_iv: "6446c0563b90e4e9e3d8db1f9e0ce859"
server_key_version: "encryptionkey@0"
I20260619 12:35:42.278875 4992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:42.283596 4992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:42.284210 4992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:42.284399 4992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:42.291185 4992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33715
I20260619 12:35:42.291193 5058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33715 every 8 connection(s)
I20260619 12:35:42.292228 4992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:42.294301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 4992
I20260619 12:35:42.294426 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:42.294713 392 external_mini_cluster.cc:1428] Setting key 4ed96d7ac265c2185734cedca52be807
I20260619 12:35:42.295969 5059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:42.300598 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.296256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48756 (local address 127.0.98.62:33715)
0619 12:35:42.296951 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:42.296965 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:42.297002 (+ 37us) server_negotiation.cc:407] Connection header received
0619 12:35:42.297657 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:42.297674 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:42.298050 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:42.298420 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:42.299264 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:42.299269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:42.299281 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:35:42.299291 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:42.299350 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:35:42.299367 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:42.299553 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:42.299630 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 12:35:42.299882 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":145,"threads_started":1}
I20260619 12:35:42.302314 5059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9: Bootstrap starting.
I20260619 12:35:42.304697 5059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:42.305473 5059 log.cc:824] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:42.307533 5059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9: No bootstrap required, opened a new log
I20260619 12:35:42.311298 5059 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8161bb05abe4df9950059a6c39b60e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33715 } }
I20260619 12:35:42.311619 5059 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:42.311702 5059 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8161bb05abe4df9950059a6c39b60e9, State: Initialized, Role: FOLLOWER
I20260619 12:35:42.312158 5059 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8161bb05abe4df9950059a6c39b60e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33715 } }
I20260619 12:35:42.312316 5059 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:42.312412 5059 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:42.312543 5059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:42.313549 5059 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8161bb05abe4df9950059a6c39b60e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33715 } }
I20260619 12:35:42.313890 5059 leader_election.cc:302] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8161bb05abe4df9950059a6c39b60e9; no voters:
I20260619 12:35:42.314234 5059 leader_election.cc:288] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:42.314435 5064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:42.314653 5064 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [term 1 LEADER]: Becoming Leader. State: Replica: c8161bb05abe4df9950059a6c39b60e9, State: Running, Role: LEADER
I20260619 12:35:42.314965 5064 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8161bb05abe4df9950059a6c39b60e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33715 } }
I20260619 12:35:42.315562 5059 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:42.316650 5066 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8161bb05abe4df9950059a6c39b60e9. Latest consensus state: current_term: 1 leader_uuid: "c8161bb05abe4df9950059a6c39b60e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8161bb05abe4df9950059a6c39b60e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33715 } } }
I20260619 12:35:42.316650 5065 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8161bb05abe4df9950059a6c39b60e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8161bb05abe4df9950059a6c39b60e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33715 } } }
I20260619 12:35:42.316798 5066 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:42.316820 5065 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:42.317170 5073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:42.320412 5073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:42.326503 5073 catalog_manager.cc:1347] Generated new cluster ID: 750cea1977504dd8876ec173d93838d2
I20260619 12:35:42.326596 5073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:42.368139 5073 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:42.368484 5073 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:42.380225 5073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9: Generated new TSK 0
I20260619 12:35:42.381107 5073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:35:42.396927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33715
--builtin_ntp_servers=127.0.98.20:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 12:35:42.510965 5083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:42.511266 5083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:42.511308 5083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:42.515529 5083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:42.515599 5083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:42.515707 5083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:35:42.520949 5083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:42.522114 5083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:42.523587 5083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:42.533707 5091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:42.533710 5088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:42.533711 5089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:42.534025 5083 server_base.cc:1034] running on GCE node
I20260619 12:35:42.534507 5083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:42.535205 5083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:42.536417 5083 hybrid_clock.cc:648] HybridClock initialized: now 1781872542536393 us; error 38 us; skew 500 ppm
I20260619 12:35:42.538509 5083 webserver.cc:460] Webserver started at http://127.0.98.1:46461/ using document root <none> and password file <none>
I20260619 12:35:42.539127 5083 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:42.539192 5083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:42.539414 5083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:42.541580 5083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "e4629732c1b244f89ae06c5cc91725fe"
format_stamp: "Formatted at 2026-06-19 12:35:42 on dist-test-slave-kdrb"
server_key: "36bd0435ba2d88ab04ac1b47a3892584"
server_key_iv: "1af1b290ab9743a8e5cf9753fe1fe6f5"
server_key_version: "encryptionkey@0"
I20260619 12:35:42.542164 5083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e4629732c1b244f89ae06c5cc91725fe"
format_stamp: "Formatted at 2026-06-19 12:35:42 on dist-test-slave-kdrb"
server_key: "36bd0435ba2d88ab04ac1b47a3892584"
server_key_iv: "1af1b290ab9743a8e5cf9753fe1fe6f5"
server_key_version: "encryptionkey@0"
I20260619 12:35:42.547320 5083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 12:35:42.550925 5097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:42.552398 5083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:35:42.552570 5083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "e4629732c1b244f89ae06c5cc91725fe"
format_stamp: "Formatted at 2026-06-19 12:35:42 on dist-test-slave-kdrb"
server_key: "36bd0435ba2d88ab04ac1b47a3892584"
server_key_iv: "1af1b290ab9743a8e5cf9753fe1fe6f5"
server_key_version: "encryptionkey@0"
I20260619 12:35:42.552706 5083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:42.557590 5083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:42.558357 5083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:42.558542 5083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:42.559082 5083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:42.560055 5083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:42.560101 5083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:42.560139 5083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:42.560154 5083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:42.568745 5083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41121
I20260619 12:35:42.568759 5210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41121 every 8 connection(s)
I20260619 12:35:42.569661 5083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:35:42.573491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 5083
I20260619 12:35:42.573637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:35:42.573982 392 external_mini_cluster.cc:1428] Setting key 1c972e1f9007a2812e86316d89a30fae
I20260619 12:35:42.576318 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33715
--builtin_ntp_servers=127.0.98.20:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 12:35:42.576372 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.571398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47517 (local address 127.0.98.62:33715)
0619 12:35:42.571580 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:42.571584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:42.572700 (+ 1116us) server_negotiation.cc:407] Connection header received
0619 12:35:42.574455 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:42.574461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:42.574509 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:42.574604 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:42.575923 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:42.575926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:42.575928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:35:42.575939 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:42.575959 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:35:42.575966 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:42.576040 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:42.576235 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 12:35:42.576269 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:35:42.577633 5213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.571870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33715 (local address 127.0.98.1:47517)
0619 12:35:42.572531 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:42.572572 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:42.574193 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:42.574808 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:42.574823 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:42.574937 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:42.575612 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:35:42.575614 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:42.575671 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:35:42.575763 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:42.576058 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:42.576070 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:42.576079 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 12:35:42.576285 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 12:35:42.576553 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":108,"threads_started":1}
I20260619 12:35:42.578958 5211 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33715
I20260619 12:35:42.579196 5211 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:42.579664 5211 heartbeater.cc:511] Master 127.0.98.62:33715 requested a full tablet report, sending...
I20260619 12:35:42.581169 5023 ts_manager.cc:194] Registered new tserver with Master: e4629732c1b244f89ae06c5cc91725fe (127.0.98.1:41121)
W20260619 12:35:42.690318 5214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:42.690645 5214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:42.690718 5214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:42.694933 5214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:42.695027 5214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:42.695122 5214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:35:42.700006 5214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:42.701159 5214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:42.702696 5214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:42.712385 5220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:42.712375 5219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:42.712419 5222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:42.712821 5214 server_base.cc:1034] running on GCE node
I20260619 12:35:42.713280 5214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:42.713977 5214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:42.715142 5214 hybrid_clock.cc:648] HybridClock initialized: now 1781872542715126 us; error 33 us; skew 500 ppm
I20260619 12:35:42.717378 5214 webserver.cc:460] Webserver started at http://127.0.98.2:39981/ using document root <none> and password file <none>
I20260619 12:35:42.718012 5214 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:42.718076 5214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:42.718297 5214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:42.720562 5214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "226b3b2819174b5698b3d1b09c13001d"
format_stamp: "Formatted at 2026-06-19 12:35:42 on dist-test-slave-kdrb"
server_key: "df442f9deb50f5e293c861538f9b58d1"
server_key_iv: "358ab90f1a91d26112f8e66ea758abbc"
server_key_version: "encryptionkey@0"
I20260619 12:35:42.721169 5214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "226b3b2819174b5698b3d1b09c13001d"
format_stamp: "Formatted at 2026-06-19 12:35:42 on dist-test-slave-kdrb"
server_key: "df442f9deb50f5e293c861538f9b58d1"
server_key_iv: "358ab90f1a91d26112f8e66ea758abbc"
server_key_version: "encryptionkey@0"
I20260619 12:35:42.725020 5214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:35:42.727628 5228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:42.728981 5214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:35:42.729125 5214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "226b3b2819174b5698b3d1b09c13001d"
format_stamp: "Formatted at 2026-06-19 12:35:42 on dist-test-slave-kdrb"
server_key: "df442f9deb50f5e293c861538f9b58d1"
server_key_iv: "358ab90f1a91d26112f8e66ea758abbc"
server_key_version: "encryptionkey@0"
I20260619 12:35:42.729251 5214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:42.734076 5214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:42.734724 5214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:42.734917 5214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:42.735563 5214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:42.736822 5214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:42.736876 5214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:42.736918 5214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:42.736990 5214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:42.745817 5214 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40123
I20260619 12:35:42.745860 5341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40123 every 8 connection(s)
I20260619 12:35:42.746810 5214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:35:42.753087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 5214
I20260619 12:35:42.753234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:35:42.753317 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.748908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48869 (local address 127.0.98.62:33715)
0619 12:35:42.749055 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:42.749059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:42.750133 (+ 1074us) server_negotiation.cc:407] Connection header received
0619 12:35:42.751524 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:42.751527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:42.751573 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:42.751687 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:42.752835 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:42.752839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:42.752842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:35:42.752856 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:42.752884 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:35:42.752893 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:42.752996 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:42.753137 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 12:35:42.753183 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:35:42.753562 392 external_mini_cluster.cc:1428] Setting key f56e05b7c17adfc8b9e24b79a5b172fb
I20260619 12:35:42.754390 5344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.749318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33715 (local address 127.0.98.2:48869)
0619 12:35:42.749975 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:42.750017 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:42.751274 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:42.751851 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:42.751858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:42.751945 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:42.752568 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:35:42.752570 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:42.752612 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:35:42.752681 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:42.752977 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:42.752985 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:42.752990 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:35:42.753170 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 12:35:42.753424 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
I20260619 12:35:42.755684 5342 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33715
I20260619 12:35:42.755806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33715
--builtin_ntp_servers=127.0.98.20:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 12:35:42.755972 5342 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:42.756400 5342 heartbeater.cc:511] Master 127.0.98.62:33715 requested a full tablet report, sending...
I20260619 12:35:42.757560 5023 ts_manager.cc:194] Registered new tserver with Master: 226b3b2819174b5698b3d1b09c13001d (127.0.98.2:40123)
W20260619 12:35:42.869207 5345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:42.869473 5345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:42.869514 5345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:42.873672 5345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:42.873735 5345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:42.873793 5345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:35:42.878525 5345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:42.879631 5345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:42.881124 5345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:42.891182 5351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:42.891323 5345 server_base.cc:1034] running on GCE node
W20260619 12:35:42.891185 5350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:42.891182 5353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:42.891948 5345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:42.892657 5345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:42.893868 5345 hybrid_clock.cc:648] HybridClock initialized: now 1781872542893849 us; error 29 us; skew 500 ppm
I20260619 12:35:42.895947 5345 webserver.cc:460] Webserver started at http://127.0.98.3:44785/ using document root <none> and password file <none>
I20260619 12:35:42.896569 5345 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:42.896667 5345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:42.896907 5345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:42.899058 5345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "d93a1ebecee441168a914cdf4fb9de0d"
format_stamp: "Formatted at 2026-06-19 12:35:42 on dist-test-slave-kdrb"
server_key: "2911261e9fc2560970d6c0652b9e721b"
server_key_iv: "de41ad8d1f2d3c87b1f2b50e8d13218b"
server_key_version: "encryptionkey@0"
I20260619 12:35:42.899722 5345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d93a1ebecee441168a914cdf4fb9de0d"
format_stamp: "Formatted at 2026-06-19 12:35:42 on dist-test-slave-kdrb"
server_key: "2911261e9fc2560970d6c0652b9e721b"
server_key_iv: "de41ad8d1f2d3c87b1f2b50e8d13218b"
server_key_version: "encryptionkey@0"
I20260619 12:35:42.903625 5345 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:35:42.906114 5359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:42.907349 5345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:35:42.907547 5345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "d93a1ebecee441168a914cdf4fb9de0d"
format_stamp: "Formatted at 2026-06-19 12:35:42 on dist-test-slave-kdrb"
server_key: "2911261e9fc2560970d6c0652b9e721b"
server_key_iv: "de41ad8d1f2d3c87b1f2b50e8d13218b"
server_key_version: "encryptionkey@0"
I20260619 12:35:42.907675 5345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:42.912364 5345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:42.913002 5345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:42.913195 5345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:42.913753 5345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:42.914669 5345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:42.914740 5345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:42.914810 5345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:42.914855 5345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:42.924276 5345 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33623
I20260619 12:35:42.924295 5472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33623 every 8 connection(s)
I20260619 12:35:42.925274 5345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:35:42.931843 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.927133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41227 (local address 127.0.98.62:33715)
0619 12:35:42.927275 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:42.927279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:42.928508 (+ 1229us) server_negotiation.cc:407] Connection header received
0619 12:35:42.929921 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:42.929925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:42.929972 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:42.930057 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:42.931301 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:42.931305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:42.931308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:35:42.931323 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:42.931352 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:35:42.931361 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:42.931469 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:42.931649 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 12:35:42.931697 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:35:42.932312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 5345
I20260619 12:35:42.932449 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:35:42.932744 392 external_mini_cluster.cc:1428] Setting key 033b0c34b5e87c235afcea4f01b45831
I20260619 12:35:42.933086 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.927654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33715 (local address 127.0.98.3:41227)
0619 12:35:42.928335 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:42.928384 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:42.929670 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:42.930259 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:42.930266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:42.930362 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:42.931021 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:35:42.931022 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:42.931066 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:35:42.931139 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:42.931458 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:42.931491 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:42.931501 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 12:35:42.931699 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 12:35:42.931974 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":110,"threads_started":1}
I20260619 12:35:42.934393 5473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33715
I20260619 12:35:42.934691 5473 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:42.935184 5473 heartbeater.cc:511] Master 127.0.98.62:33715 requested a full tablet report, sending...
I20260619 12:35:42.936367 5023 ts_manager.cc:194] Registered new tserver with Master: d93a1ebecee441168a914cdf4fb9de0d (127.0.98.3:33623)
I20260619 12:35:42.947278 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:35:42.950929 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.949379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48766 (local address 127.0.98.62:33715)
0619 12:35:42.949543 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:42.949547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:42.949603 (+ 56us) server_negotiation.cc:407] Connection header received
0619 12:35:42.949725 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:42.949727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:42.949759 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:42.949826 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:42.950572 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:42.950575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:42.950577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:35:42.950584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:42.950599 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:35:42.950606 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:42.950670 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:42.950717 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 12:35:42.950748 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 12:35:42.954036 5023 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:35:42.957234 5023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:42.969219 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.966799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60350 (local address 127.0.98.3:33623)
0619 12:35:42.967174 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:42.967187 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:42.967214 (+ 27us) server_negotiation.cc:407] Connection header received
0619 12:35:42.967267 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:42.967273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:42.967498 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:42.967889 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:42.968694 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:42.968698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:42.968706 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:35:42.968713 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:42.968741 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:35:42.968755 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:42.968876 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:42.968985 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 12:35:42.969053 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":195,"threads_started":1}
I20260619 12:35:42.969210 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.965785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33623 (local address 127.0.0.1:60350)
0619 12:35:42.966741 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:42.966781 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:42.966895 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:42.967636 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:42.967639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:42.967653 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:42.968266 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:35:42.968268 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:42.968288 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:35:42.968353 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:42.968849 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:42.968858 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:42.968863 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:35:42.968973 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 12:35:42.969042 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"spinlock_wait_cycles":6912,"thread_start_us":126,"threads_started":1}
I20260619 12:35:42.969547 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.965922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45936 (local address 127.0.98.1:41121)
0619 12:35:42.966263 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:42.966279 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:42.966901 (+ 622us) server_negotiation.cc:407] Connection header received
0619 12:35:42.967015 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:42.967023 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:42.967234 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:42.968085 (+ 851us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:42.968764 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:42.968768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:42.968776 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:35:42.968784 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:42.968814 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:35:42.968828 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:42.968959 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:42.969271 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 12:35:42.969347 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":148,"threads_started":1}
I20260619 12:35:42.969681 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.966443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40123 (local address 127.0.0.1:38998)
0619 12:35:42.967271 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:42.967283 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:42.967355 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:42.967847 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:42.967851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:42.967875 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:42.969044 (+ 1169us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:35:42.969046 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:42.969055 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:35:42.969076 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:42.969447 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:42.969452 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:42.969455 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:35:42.969533 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 12:35:42.969569 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":98,"threads_started":1}
I20260619 12:35:42.969817 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.966578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38998 (local address 127.0.98.2:40123)
0619 12:35:42.967145 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:42.967156 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:42.967452 (+ 296us) server_negotiation.cc:407] Connection header received
0619 12:35:42.967533 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:42.967542 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:42.967700 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:42.967841 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:42.969201 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:42.969207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:42.969217 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:35:42.969230 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:42.969273 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:35:42.969304 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:42.969402 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:42.969582 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 12:35:42.969650 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":103,"threads_started":1}
I20260619 12:35:42.969219 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:42.965772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41121 (local address 127.0.0.1:45936)
0619 12:35:42.966726 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:42.966779 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:42.966910 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:42.967427 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:42.967433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:42.967552 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:42.968559 (+ 1007us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:35:42.968561 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:42.968572 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:35:42.968598 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:42.968931 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:42.968934 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:42.968937 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:35:42.969022 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 12:35:42.969059 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"spinlock_wait_cycles":69504,"thread_start_us":93,"threads_started":1}
I20260619 12:35:42.972615 5145 tablet_service.cc:1467] Processing CreateTablet for tablet 061e7d03fc2942a29a356f0478b9f5e0 (DEFAULT_TABLE table=test-table [id=fdd0619613df4e84806812c1b322d255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:42.972702 5407 tablet_service.cc:1467] Processing CreateTablet for tablet 061e7d03fc2942a29a356f0478b9f5e0 (DEFAULT_TABLE table=test-table [id=fdd0619613df4e84806812c1b322d255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:42.972705 5276 tablet_service.cc:1467] Processing CreateTablet for tablet 061e7d03fc2942a29a356f0478b9f5e0 (DEFAULT_TABLE table=test-table [id=fdd0619613df4e84806812c1b322d255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:42.973676 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 061e7d03fc2942a29a356f0478b9f5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:42.973676 5407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 061e7d03fc2942a29a356f0478b9f5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:42.973678 5145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 061e7d03fc2942a29a356f0478b9f5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:42.979719 5490 tablet_bootstrap.cc:492] T 061e7d03fc2942a29a356f0478b9f5e0 P e4629732c1b244f89ae06c5cc91725fe: Bootstrap starting.
I20260619 12:35:42.981469 5492 tablet_bootstrap.cc:492] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d: Bootstrap starting.
I20260619 12:35:42.981979 5490 tablet_bootstrap.cc:654] T 061e7d03fc2942a29a356f0478b9f5e0 P e4629732c1b244f89ae06c5cc91725fe: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:42.982897 5490 log.cc:824] T 061e7d03fc2942a29a356f0478b9f5e0 P e4629732c1b244f89ae06c5cc91725fe: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:42.983671 5491 tablet_bootstrap.cc:492] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d: Bootstrap starting.
I20260619 12:35:42.983732 5492 tablet_bootstrap.cc:654] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:42.984416 5492 log.cc:824] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:42.985126 5490 tablet_bootstrap.cc:492] T 061e7d03fc2942a29a356f0478b9f5e0 P e4629732c1b244f89ae06c5cc91725fe: No bootstrap required, opened a new log
I20260619 12:35:42.985321 5490 ts_tablet_manager.cc:1397] T 061e7d03fc2942a29a356f0478b9f5e0 P e4629732c1b244f89ae06c5cc91725fe: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:35:42.985764 5491 tablet_bootstrap.cc:654] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:42.986258 5492 tablet_bootstrap.cc:492] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d: No bootstrap required, opened a new log
I20260619 12:35:42.986459 5492 ts_tablet_manager.cc:1397] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:35:42.986548 5491 log.cc:824] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:42.988693 5491 tablet_bootstrap.cc:492] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d: No bootstrap required, opened a new log
I20260619 12:35:42.988890 5491 ts_tablet_manager.cc:1397] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:35:42.989966 5490 raft_consensus.cc:348] T 061e7d03fc2942a29a356f0478b9f5e0 P e4629732c1b244f89ae06c5cc91725fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4629732c1b244f89ae06c5cc91725fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } } peers { permanent_uuid: "d93a1ebecee441168a914cdf4fb9de0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33623 } } peers { permanent_uuid: "226b3b2819174b5698b3d1b09c13001d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40123 } }
I20260619 12:35:42.990265 5490 raft_consensus.cc:374] T 061e7d03fc2942a29a356f0478b9f5e0 P e4629732c1b244f89ae06c5cc91725fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:42.990329 5490 raft_consensus.cc:729] T 061e7d03fc2942a29a356f0478b9f5e0 P e4629732c1b244f89ae06c5cc91725fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4629732c1b244f89ae06c5cc91725fe, State: Initialized, Role: FOLLOWER
I20260619 12:35:42.990516 5492 raft_consensus.cc:348] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4629732c1b244f89ae06c5cc91725fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } } peers { permanent_uuid: "d93a1ebecee441168a914cdf4fb9de0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33623 } } peers { permanent_uuid: "226b3b2819174b5698b3d1b09c13001d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40123 } }
I20260619 12:35:42.990803 5492 raft_consensus.cc:374] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:42.990882 5492 raft_consensus.cc:729] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93a1ebecee441168a914cdf4fb9de0d, State: Initialized, Role: FOLLOWER
I20260619 12:35:42.990809 5490 consensus_queue.cc:260] T 061e7d03fc2942a29a356f0478b9f5e0 P e4629732c1b244f89ae06c5cc91725fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4629732c1b244f89ae06c5cc91725fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } } peers { permanent_uuid: "d93a1ebecee441168a914cdf4fb9de0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33623 } } peers { permanent_uuid: "226b3b2819174b5698b3d1b09c13001d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40123 } }
I20260619 12:35:42.991310 5492 consensus_queue.cc:260] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4629732c1b244f89ae06c5cc91725fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } } peers { permanent_uuid: "d93a1ebecee441168a914cdf4fb9de0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33623 } } peers { permanent_uuid: "226b3b2819174b5698b3d1b09c13001d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40123 } }
I20260619 12:35:42.992018 5473 heartbeater.cc:503] Master 127.0.98.62:33715 was elected leader, sending a full tablet report...
I20260619 12:35:42.992027 5490 ts_tablet_manager.cc:1428] T 061e7d03fc2942a29a356f0478b9f5e0 P e4629732c1b244f89ae06c5cc91725fe: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 12:35:42.992312 5492 ts_tablet_manager.cc:1428] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 12:35:42.992348 5211 heartbeater.cc:503] Master 127.0.98.62:33715 was elected leader, sending a full tablet report...
I20260619 12:35:42.993319 5491 raft_consensus.cc:348] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4629732c1b244f89ae06c5cc91725fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } } peers { permanent_uuid: "d93a1ebecee441168a914cdf4fb9de0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33623 } } peers { permanent_uuid: "226b3b2819174b5698b3d1b09c13001d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40123 } }
I20260619 12:35:42.993628 5491 raft_consensus.cc:374] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:42.993700 5491 raft_consensus.cc:729] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226b3b2819174b5698b3d1b09c13001d, State: Initialized, Role: FOLLOWER
I20260619 12:35:42.994133 5491 consensus_queue.cc:260] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4629732c1b244f89ae06c5cc91725fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } } peers { permanent_uuid: "d93a1ebecee441168a914cdf4fb9de0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33623 } } peers { permanent_uuid: "226b3b2819174b5698b3d1b09c13001d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40123 } }
I20260619 12:35:42.994745 5342 heartbeater.cc:503] Master 127.0.98.62:33715 was elected leader, sending a full tablet report...
I20260619 12:35:42.995122 5491 ts_tablet_manager.cc:1428] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
W20260619 12:35:42.997833 5343 tablet.cc:2369] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:35:43.070736 5212 tablet.cc:2369] T 061e7d03fc2942a29a356f0478b9f5e0 P e4629732c1b244f89ae06c5cc91725fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:35:43.176308 5474 tablet.cc:2369] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:35:43.184736 5498 raft_consensus.cc:482] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:43.184973 5498 raft_consensus.cc:504] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4629732c1b244f89ae06c5cc91725fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } } peers { permanent_uuid: "d93a1ebecee441168a914cdf4fb9de0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33623 } } peers { permanent_uuid: "226b3b2819174b5698b3d1b09c13001d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40123 } }
I20260619 12:35:43.186174 5498 leader_election.cc:288] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4629732c1b244f89ae06c5cc91725fe (127.0.98.1:41121), d93a1ebecee441168a914cdf4fb9de0d (127.0.98.3:33623)
I20260619 12:35:43.188263 5344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:43.186447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41121 (local address 127.0.98.2:57007)
0619 12:35:43.186579 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:43.186597 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:43.186692 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:43.187029 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:43.187032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:43.187047 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:43.187660 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:35:43.187664 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:43.187680 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:35:43.187714 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:43.187988 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:43.187991 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:43.187994 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:35:43.188077 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 12:35:43.188116 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 12:35:43.188249 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:43.186671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57007 (local address 127.0.98.1:41121)
0619 12:35:43.186797 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:43.186800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:43.186812 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:35:43.186853 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:43.186855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:43.186886 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:43.187004 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:43.187854 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:43.187857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:43.187859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:35:43.187870 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:43.187889 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:35:43.187897 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:43.187994 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:43.188079 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 12:35:43.188116 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:35:43.188560 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:43.186581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55051 (local address 127.0.98.3:33623)
0619 12:35:43.186713 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:43.186719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:43.187064 (+ 345us) server_negotiation.cc:407] Connection header received
0619 12:35:43.187159 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:43.187162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:43.187234 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:43.187322 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:43.188137 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:43.188140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:43.188145 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:35:43.188162 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:43.188181 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:35:43.188188 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:43.188284 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:43.188395 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 12:35:43.188445 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:35:43.188901 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:43.186448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33623 (local address 127.0.98.2:55051)
0619 12:35:43.186957 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:43.186969 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:43.187073 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:43.187395 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:43.187397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:43.187410 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:43.187968 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:35:43.187969 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:43.187979 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:35:43.188002 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:43.188277 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:43.188280 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:43.188283 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:35:43.188722 (+ 439us) client_negotiation.cc:240] Negotiation successful
0619 12:35:43.188761 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":147,"thread_start_us":148,"threads_started":1}
I20260619 12:35:43.189014 5165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "061e7d03fc2942a29a356f0478b9f5e0" candidate_uuid: "226b3b2819174b5698b3d1b09c13001d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4629732c1b244f89ae06c5cc91725fe" is_pre_election: true
I20260619 12:35:43.189404 5165 raft_consensus.cc:2454] T 061e7d03fc2942a29a356f0478b9f5e0 P e4629732c1b244f89ae06c5cc91725fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 226b3b2819174b5698b3d1b09c13001d in term 0.
I20260619 12:35:43.189565 5427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "061e7d03fc2942a29a356f0478b9f5e0" candidate_uuid: "226b3b2819174b5698b3d1b09c13001d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93a1ebecee441168a914cdf4fb9de0d" is_pre_election: true
I20260619 12:35:43.189900 5427 raft_consensus.cc:2454] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 226b3b2819174b5698b3d1b09c13001d in term 0.
I20260619 12:35:43.189917 5229 leader_election.cc:302] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226b3b2819174b5698b3d1b09c13001d, e4629732c1b244f89ae06c5cc91725fe; no voters:
I20260619 12:35:43.190251 5498 raft_consensus.cc:2781] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:43.190353 5498 raft_consensus.cc:482] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:43.190398 5498 raft_consensus.cc:3037] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:43.191531 5498 raft_consensus.cc:504] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4629732c1b244f89ae06c5cc91725fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } } peers { permanent_uuid: "d93a1ebecee441168a914cdf4fb9de0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33623 } } peers { permanent_uuid: "226b3b2819174b5698b3d1b09c13001d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40123 } }
I20260619 12:35:43.191897 5498 leader_election.cc:288] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [CANDIDATE]: Term 1 election: Requested vote from peers e4629732c1b244f89ae06c5cc91725fe (127.0.98.1:41121), d93a1ebecee441168a914cdf4fb9de0d (127.0.98.3:33623)
I20260619 12:35:43.192256 5427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "061e7d03fc2942a29a356f0478b9f5e0" candidate_uuid: "226b3b2819174b5698b3d1b09c13001d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93a1ebecee441168a914cdf4fb9de0d"
I20260619 12:35:43.192266 5165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "061e7d03fc2942a29a356f0478b9f5e0" candidate_uuid: "226b3b2819174b5698b3d1b09c13001d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4629732c1b244f89ae06c5cc91725fe"
I20260619 12:35:43.192369 5427 raft_consensus.cc:3037] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:43.192404 5165 raft_consensus.cc:3037] T 061e7d03fc2942a29a356f0478b9f5e0 P e4629732c1b244f89ae06c5cc91725fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:43.193493 5427 raft_consensus.cc:2454] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 226b3b2819174b5698b3d1b09c13001d in term 1.
I20260619 12:35:43.193502 5165 raft_consensus.cc:2454] T 061e7d03fc2942a29a356f0478b9f5e0 P e4629732c1b244f89ae06c5cc91725fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 226b3b2819174b5698b3d1b09c13001d in term 1.
I20260619 12:35:43.193810 5229 leader_election.cc:302] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226b3b2819174b5698b3d1b09c13001d, e4629732c1b244f89ae06c5cc91725fe; no voters:
I20260619 12:35:43.194010 5498 raft_consensus.cc:2781] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:43.194249 5498 raft_consensus.cc:686] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [term 1 LEADER]: Becoming Leader. State: Replica: 226b3b2819174b5698b3d1b09c13001d, State: Running, Role: LEADER
I20260619 12:35:43.194619 5498 consensus_queue.cc:237] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4629732c1b244f89ae06c5cc91725fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } } peers { permanent_uuid: "d93a1ebecee441168a914cdf4fb9de0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33623 } } peers { permanent_uuid: "226b3b2819174b5698b3d1b09c13001d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40123 } }
I20260619 12:35:43.198009 5020 catalog_manager.cc:5495] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d reported cstate change: term changed from 0 to 1, leader changed from <none> to 226b3b2819174b5698b3d1b09c13001d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "226b3b2819174b5698b3d1b09c13001d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4629732c1b244f89ae06c5cc91725fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d93a1ebecee441168a914cdf4fb9de0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "226b3b2819174b5698b3d1b09c13001d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40123 } health_report { overall_health: HEALTHY } } }
I20260619 12:35:43.263201 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:43.261712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.0.98.2:40123)
0619 12:35:43.261864 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:43.261869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:43.261889 (+ 20us) server_negotiation.cc:407] Connection header received
0619 12:35:43.261968 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:43.261970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:43.262027 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:43.262120 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:43.262776 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:43.262780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:43.262782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:35:43.262794 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:43.262819 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:35:43.262828 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:43.262924 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:43.263013 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 12:35:43.263055 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:35:43.269042 5165 raft_consensus.cc:1261] T 061e7d03fc2942a29a356f0478b9f5e0 P e4629732c1b244f89ae06c5cc91725fe [term 1 FOLLOWER]: Refusing update from remote peer 226b3b2819174b5698b3d1b09c13001d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:43.269047 5427 raft_consensus.cc:1261] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d [term 1 FOLLOWER]: Refusing update from remote peer 226b3b2819174b5698b3d1b09c13001d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:43.269809 5500 consensus_queue.cc:1035] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93a1ebecee441168a914cdf4fb9de0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:43.270037 5498 consensus_queue.cc:1035] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4629732c1b244f89ae06c5cc91725fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:43.286335 5023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:43.286556 5023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:43.289224 5023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8161bb05abe4df9950059a6c39b60e9: Sending DeleteTablet for 3 replicas of tablet 061e7d03fc2942a29a356f0478b9f5e0
I20260619 12:35:43.290153 5276 tablet_service.cc:1514] Processing DeleteTablet for tablet 061e7d03fc2942a29a356f0478b9f5e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:43 UTC) from {username='slave'} at 127.0.0.1:38998
I20260619 12:35:43.290148 5145 tablet_service.cc:1514] Processing DeleteTablet for tablet 061e7d03fc2942a29a356f0478b9f5e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:43 UTC) from {username='slave'} at 127.0.0.1:45936
I20260619 12:35:43.290709 5513 tablet_replica.cc:330] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d: stopping tablet replica
I20260619 12:35:43.290704 5407 tablet_service.cc:1514] Processing DeleteTablet for tablet 061e7d03fc2942a29a356f0478b9f5e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:43 UTC) from {username='slave'} at 127.0.0.1:60350
I20260619 12:35:43.291040 5514 tablet_replica.cc:330] T 061e7d03fc2942a29a356f0478b9f5e0 P e4629732c1b244f89ae06c5cc91725fe: stopping tablet replica
I20260619 12:35:43.291455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 5083
I20260619 12:35:43.291798 5513 raft_consensus.cc:2229] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:35:43.292397 5513 raft_consensus.cc:2258] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:43.294004 5513 ts_tablet_manager.cc:1905] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:43.297134 5513 ts_tablet_manager.cc:1918] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:43.297333 5513 log.cc:1196] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/061e7d03fc2942a29a356f0478b9f5e0
I20260619 12:35:43.297778 5513 ts_tablet_manager.cc:1939] T 061e7d03fc2942a29a356f0478b9f5e0 P 226b3b2819174b5698b3d1b09c13001d: Deleting consensus metadata
I20260619 12:35:43.299168 5010 catalog_manager.cc:4841] TS 226b3b2819174b5698b3d1b09c13001d (127.0.98.2:40123): tablet 061e7d03fc2942a29a356f0478b9f5e0 (table test-table [id=fdd0619613df4e84806812c1b322d255]) successfully deleted
W20260619 12:35:43.301354 5007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41121 (error 108)
I20260619 12:35:43.302428 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:43.302114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41121 (local address 127.0.0.1:45948)
0619 12:35:43.302261 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:43.302340 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 12:35:43.302444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 5214
I20260619 12:35:43.303188 5515 tablet_replica.cc:330] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d: stopping tablet replica
I20260619 12:35:43.303601 5515 raft_consensus.cc:2229] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:43.303984 5515 raft_consensus.cc:2258] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 12:35:43.304529 5007 catalog_manager.cc:4584] TS e4629732c1b244f89ae06c5cc91725fe (127.0.98.1:41121): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 061e7d03fc2942a29a356f0478b9f5e0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41121: connect: Connection refused (error 111)
I20260619 12:35:43.306411 5515 ts_tablet_manager.cc:1905] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:43.310197 5515 ts_tablet_manager.cc:1918] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:43.310320 5515 log.cc:1196] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/061e7d03fc2942a29a356f0478b9f5e0
I20260619 12:35:43.310686 5515 ts_tablet_manager.cc:1939] T 061e7d03fc2942a29a356f0478b9f5e0 P d93a1ebecee441168a914cdf4fb9de0d: Deleting consensus metadata
I20260619 12:35:43.310992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 5345
I20260619 12:35:43.317865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 4992
2026-06-19T12:35:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1219 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 12:35:43.339980 392 mini_oidc.cc:152] Starting JWKS server
I20260619 12:35:43.341527 392 webserver.cc:460] Webserver started at https://127.0.0.1:41971/ using document root <none> and password file <none>
I20260619 12:35:43.341670 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 12:35:43.341890 392 webserver.cc:460] Webserver started at http://127.0.0.1:37285/ using document root <none> and password file <none>
2026-06-19T12:35:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:43Z Disabled control of system clock
I20260619 12:35:43.350807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46025
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46025
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41971/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/testchainca.cert with env {}
W20260619 12:35:43.461615 5522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:43.461910 5522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:43.461952 5522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:43.466284 5522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:43.466354 5522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:43.466370 5522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:43.466387 5522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:43.466403 5522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 12:35:43.466418 5522 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 12:35:43.466432 5522 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41971/jwks.json
I20260619 12:35:43.472033 5522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46025
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41971/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:43.473119 5522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:43.474128 5522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:43.483062 5527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:43.483100 5528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:43.483227 5522 server_base.cc:1034] running on GCE node
W20260619 12:35:43.483110 5530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:43.484022 5522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:43.485162 5522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:43.486326 5522 hybrid_clock.cc:648] HybridClock initialized: now 1781872543486300 us; error 42 us; skew 500 ppm
I20260619 12:35:43.488265 5522 webserver.cc:460] Webserver started at http://127.0.98.62:44595/ using document root <none> and password file <none>
I20260619 12:35:43.488857 5522 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:43.488922 5522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:43.489154 5522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:43.491223 5522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "13f1784a11694684aa52ac1894676c39"
format_stamp: "Formatted at 2026-06-19 12:35:43 on dist-test-slave-kdrb"
server_key: "9dd81862f74d511e7469cff099715d4a"
server_key_iv: "1522d5b5bc1eaeee71f91a37e637ec31"
server_key_version: "encryptionkey@0"
I20260619 12:35:43.491871 5522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "13f1784a11694684aa52ac1894676c39"
format_stamp: "Formatted at 2026-06-19 12:35:43 on dist-test-slave-kdrb"
server_key: "9dd81862f74d511e7469cff099715d4a"
server_key_iv: "1522d5b5bc1eaeee71f91a37e637ec31"
server_key_version: "encryptionkey@0"
I20260619 12:35:43.495675 5522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:35:43.498165 5536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:43.499538 5522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:35:43.499670 5522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "13f1784a11694684aa52ac1894676c39"
format_stamp: "Formatted at 2026-06-19 12:35:43 on dist-test-slave-kdrb"
server_key: "9dd81862f74d511e7469cff099715d4a"
server_key_iv: "1522d5b5bc1eaeee71f91a37e637ec31"
server_key_version: "encryptionkey@0"
I20260619 12:35:43.499789 5522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:43.516155 5522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:43.516891 5522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:43.517084 5522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:43.524194 5522 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46025
I20260619 12:35:43.524204 5588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46025 every 8 connection(s)
I20260619 12:35:43.525178 5522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:43.526463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 5522
I20260619 12:35:43.526561 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:43.526846 392 external_mini_cluster.cc:1428] Setting key b7f23248dd677b345e43e5dab35b7760
I20260619 12:35:43.528848 5589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:43.535408 5589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39: Bootstrap starting.
I20260619 12:35:43.536175 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:43.528185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.0.98.62:46025)
0619 12:35:43.528782 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:43.528798 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:43.528833 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:35:43.529530 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:43.529547 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:43.529933 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:43.530215 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:43.530550 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:43.531563 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:43.532787 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:43.533243 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:43.533358 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:43.534260 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:43.534264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:43.534277 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:35:43.534685 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:43.534744 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:35:43.534763 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:43.534838 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:43.535006 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 12:35:43.535312 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":127,"threads_started":1}
I20260619 12:35:43.537957 5589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:43.538859 5589 log.cc:824] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:43.539224 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:35:43.541131 5589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39: No bootstrap required, opened a new log
I20260619 12:35:43.544754 5589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f1784a11694684aa52ac1894676c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46025 } }
I20260619 12:35:43.544987 5589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:43.545022 5589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f1784a11694684aa52ac1894676c39, State: Initialized, Role: FOLLOWER
I20260619 12:35:43.545439 5589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f1784a11694684aa52ac1894676c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46025 } }
I20260619 12:35:43.545558 5589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:43.545604 5589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:43.545674 5589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:43.546638 5589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f1784a11694684aa52ac1894676c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46025 } }
I20260619 12:35:43.547009 5589 leader_election.cc:302] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13f1784a11694684aa52ac1894676c39; no voters:
I20260619 12:35:43.547364 5589 leader_election.cc:288] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:43.547509 5594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:43.547766 5594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [term 1 LEADER]: Becoming Leader. State: Replica: 13f1784a11694684aa52ac1894676c39, State: Running, Role: LEADER
I20260619 12:35:43.548080 5594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f1784a11694684aa52ac1894676c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46025 } }
I20260619 12:35:43.548602 5589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:43.549649 5595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13f1784a11694684aa52ac1894676c39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f1784a11694684aa52ac1894676c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46025 } } }
I20260619 12:35:43.549778 5595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:43.550153 5596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13f1784a11694684aa52ac1894676c39. Latest consensus state: current_term: 1 leader_uuid: "13f1784a11694684aa52ac1894676c39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f1784a11694684aa52ac1894676c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46025 } } }
I20260619 12:35:43.550254 5596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:43.550141 5603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:43.553484 5603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:43.559273 5603 catalog_manager.cc:1347] Generated new cluster ID: b7bbe3a800664abe9e47ec8829d67768
I20260619 12:35:43.559351 5603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:43.579064 5603 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:43.580408 5603 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:43.586123 5603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13f1784a11694684aa52ac1894676c39: Generated new TSK 0
I20260619 12:35:43.586988 5603 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 12:35:43.618702 5619 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 12:35:43.618840 5619 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 12:35:43.619556 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:43.602261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.0.98.62:46025)
0619 12:35:43.602429 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:43.602433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:43.602660 (+ 227us) server_negotiation.cc:407] Connection header received
0619 12:35:43.602923 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:43.602928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:43.602988 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:43.603079 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0619 12:35:43.603382 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:43.604011 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:43.605288 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:43.605509 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:43.605593 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:43.605992 (+ 399us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 12:35:43.619144 (+ 13152us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 12:35:43.619377 (+ 233us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34614: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 12:35:43.619639 5592 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34614: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 12:35:43.621060 5618 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:35:43.602183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46025 (local address 127.0.0.1:34614)
0619 12:35:43.602557 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:43.602573 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:43.602721 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:43.603123 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:43.603126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:43.603135 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 12:35:43.603268 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:43.603275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:43.604132 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:43.604136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:43.605157 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:43.605169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:43.605609 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:43.605636 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:43.605733 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:43.605801 (+ 68us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 12:35:43.619385 (+ 13584us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 12:35:43.619685 (+ 300us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46025: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":129,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:46025: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 12:35:43.621865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 5522
2026-06-19T12:35:43Z chronyd exiting
I20260619 12:35:43.659650 392 test_util.cc:175] -----------------------------------------------
I20260619 12:35:43.659745 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872516972220-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (321 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 12:35:43.660776 392 mini_oidc.cc:152] Starting JWKS server
I20260619 12:35:43.661607 392 webserver.cc:460] Webserver started at https://127.0.0.1:34857/ using document root <none> and password file <none>
I20260619 12:35:43.661692 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 12:35:43.661917 392 webserver.cc:460] Webserver started at http://127.0.0.1:35861/ using document root <none> and password file <none>
2026-06-19T12:35:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:43Z Disabled control of system clock
I20260619 12:35:43.671378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38611
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38611
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34857/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/testchainca.cert with env {}
W20260619 12:35:43.781720 5625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:43.782095 5625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:43.782182 5625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:43.786378 5625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:43.786490 5625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:43.786518 5625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:43.786557 5625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:43.786614 5625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 12:35:43.786656 5625 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 12:35:43.786688 5625 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34857/jwks.json
I20260619 12:35:43.792330 5625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38611
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34857/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:43.793541 5625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:43.794633 5625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:43.803279 5631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:43.803329 5630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:43.803432 5625 server_base.cc:1034] running on GCE node
W20260619 12:35:43.803653 5633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:43.804260 5625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:43.805440 5625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:43.806635 5625 hybrid_clock.cc:648] HybridClock initialized: now 1781872543806619 us; error 34 us; skew 500 ppm
I20260619 12:35:43.808732 5625 webserver.cc:460] Webserver started at http://127.0.98.62:39523/ using document root <none> and password file <none>
I20260619 12:35:43.809343 5625 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:43.809439 5625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:43.809669 5625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:43.811821 5625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "d7da70fd9b6c421c920203ff504ad8c3"
format_stamp: "Formatted at 2026-06-19 12:35:43 on dist-test-slave-kdrb"
server_key: "d2fa2753c292ec220eb7dff4615084d1"
server_key_iv: "d573827ebba8a0a4fd8e96a53bc127ed"
server_key_version: "encryptionkey@0"
I20260619 12:35:43.812410 5625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7da70fd9b6c421c920203ff504ad8c3"
format_stamp: "Formatted at 2026-06-19 12:35:43 on dist-test-slave-kdrb"
server_key: "d2fa2753c292ec220eb7dff4615084d1"
server_key_iv: "d573827ebba8a0a4fd8e96a53bc127ed"
server_key_version: "encryptionkey@0"
I20260619 12:35:43.816421 5625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 12:35:43.818966 5639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:43.820348 5625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 12:35:43.820503 5625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "d7da70fd9b6c421c920203ff504ad8c3"
format_stamp: "Formatted at 2026-06-19 12:35:43 on dist-test-slave-kdrb"
server_key: "d2fa2753c292ec220eb7dff4615084d1"
server_key_iv: "d573827ebba8a0a4fd8e96a53bc127ed"
server_key_version: "encryptionkey@0"
I20260619 12:35:43.820629 5625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:43.835887 5625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:43.836659 5625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:43.836876 5625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:43.843924 5625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38611
I20260619 12:35:43.843928 5691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38611 every 8 connection(s)
I20260619 12:35:43.844933 5625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:43.846998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 5625
I20260619 12:35:43.847141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:43.847424 392 external_mini_cluster.cc:1428] Setting key f8d00d79e8b8c608249df5de4b7aaefb
I20260619 12:35:43.848691 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:43.854274 5692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3: Bootstrap starting.
I20260619 12:35:43.856648 5692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:43.856745 5695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:43.848804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43590 (local address 127.0.98.62:38611)
0619 12:35:43.849465 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:43.849481 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:43.849517 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:35:43.850167 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:43.850183 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:43.850590 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:43.850941 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:43.851378 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:43.852423 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:43.853352 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:43.853841 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:43.854015 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:43.854964 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:43.854969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:43.854982 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:35:43.855383 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:43.855444 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:35:43.855465 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:43.855625 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:43.855745 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 12:35:43.856024 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":137,"threads_started":1}
I20260619 12:35:43.857471 5692 log.cc:824] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:43.859685 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:35:43.859697 5692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3: No bootstrap required, opened a new log
I20260619 12:35:43.863453 5692 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7da70fd9b6c421c920203ff504ad8c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38611 } }
I20260619 12:35:43.863797 5692 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:43.863880 5692 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7da70fd9b6c421c920203ff504ad8c3, State: Initialized, Role: FOLLOWER
I20260619 12:35:43.864326 5692 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7da70fd9b6c421c920203ff504ad8c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38611 } }
I20260619 12:35:43.864497 5692 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:43.864584 5692 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:43.864699 5692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:43.865655 5692 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7da70fd9b6c421c920203ff504ad8c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38611 } }
I20260619 12:35:43.866074 5692 leader_election.cc:302] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7da70fd9b6c421c920203ff504ad8c3; no voters:
I20260619 12:35:43.866477 5692 leader_election.cc:288] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:43.866595 5697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:43.866847 5697 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [term 1 LEADER]: Becoming Leader. State: Replica: d7da70fd9b6c421c920203ff504ad8c3, State: Running, Role: LEADER
I20260619 12:35:43.867166 5697 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7da70fd9b6c421c920203ff504ad8c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38611 } }
I20260619 12:35:43.867893 5692 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:43.869002 5699 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7da70fd9b6c421c920203ff504ad8c3. Latest consensus state: current_term: 1 leader_uuid: "d7da70fd9b6c421c920203ff504ad8c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7da70fd9b6c421c920203ff504ad8c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38611 } } }
I20260619 12:35:43.869148 5699 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:43.869422 5698 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7da70fd9b6c421c920203ff504ad8c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7da70fd9b6c421c920203ff504ad8c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38611 } } }
I20260619 12:35:43.869511 5698 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:43.869632 5706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:43.872478 5706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:43.878610 5706 catalog_manager.cc:1347] Generated new cluster ID: e31f528797f146e9bffa19eab7d5bd43
I20260619 12:35:43.878697 5706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:43.893687 5706 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:43.895013 5706 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:43.903462 5706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7da70fd9b6c421c920203ff504ad8c3: Generated new TSK 0
I20260619 12:35:43.904386 5706 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 12:35:43.931238 5722 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 12:35:43.931346 5722 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 12:35:43.932025 5695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:43.915719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43602 (local address 127.0.98.62:38611)
0619 12:35:43.915859 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:43.915864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:43.916035 (+ 171us) server_negotiation.cc:407] Connection header received
0619 12:35:43.916194 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:43.916196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:43.916244 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:43.916339 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0619 12:35:43.916587 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:43.917123 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:43.918361 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:43.918586 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:43.918665 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:43.918990 (+ 325us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 12:35:43.931675 (+ 12685us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34857/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 12:35:43.931912 (+ 237us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43602: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34857/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 12:35:43.932108 5695 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43602: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34857/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 12:35:43.932277 5721 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:35:43.915646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38611 (local address 127.0.0.1:43602)
0619 12:35:43.915941 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:43.915956 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:43.916075 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:43.916361 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:43.916364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:43.916372 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 12:35:43.916485 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:43.916492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:43.917232 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:43.917236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:43.918240 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:43.918252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:43.918683 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:43.918705 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:43.918796 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:43.918829 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 12:35:43.931879 (+ 13050us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34857/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 12:35:43.932093 (+ 214us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38611: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34857/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":113,"threads_started":1}
I20260619 12:35:43.932914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 5625
2026-06-19T12:35:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (311 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 12:35:43.972262 392 mini_oidc.cc:152] Starting JWKS server
I20260619 12:35:43.973156 392 webserver.cc:460] Webserver started at https://127.0.0.1:33359/ using document root <none> and password file <none>
I20260619 12:35:43.973275 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 12:35:43.973567 392 webserver.cc:460] Webserver started at http://127.0.0.1:46417/ using document root <none> and password file <none>
2026-06-19T12:35:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:43Z Disabled control of system clock
I20260619 12:35:43.982743 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38601
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38601
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33359/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 12:35:44.094429 5728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:44.094722 5728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:44.094762 5728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:44.098873 5728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:44.098940 5728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:44.098956 5728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:44.098974 5728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:44.098990 5728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 12:35:44.099012 5728 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 12:35:44.099032 5728 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33359/jwks.json
I20260619 12:35:44.103822 5728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33359/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:44.104912 5728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:44.106261 5728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:44.117254 5734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:44.117292 5728 server_base.cc:1034] running on GCE node
W20260619 12:35:44.117513 5733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:44.117251 5736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:44.118405 5728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:44.119647 5728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:44.120826 5728 hybrid_clock.cc:648] HybridClock initialized: now 1781872544120798 us; error 44 us; skew 500 ppm
I20260619 12:35:44.122821 5728 webserver.cc:460] Webserver started at http://127.0.98.62:41005/ using document root <none> and password file <none>
I20260619 12:35:44.123399 5728 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:44.123463 5728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:44.123733 5728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:44.125857 5728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "e53017687a184a2a94e18aeb3c7d5728"
format_stamp: "Formatted at 2026-06-19 12:35:44 on dist-test-slave-kdrb"
server_key: "4531d15a6b400a07b432712b78754a21"
server_key_iv: "238dea30613f63d29efcf6335bb5e375"
server_key_version: "encryptionkey@0"
I20260619 12:35:44.126437 5728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "e53017687a184a2a94e18aeb3c7d5728"
format_stamp: "Formatted at 2026-06-19 12:35:44 on dist-test-slave-kdrb"
server_key: "4531d15a6b400a07b432712b78754a21"
server_key_iv: "238dea30613f63d29efcf6335bb5e375"
server_key_version: "encryptionkey@0"
I20260619 12:35:44.130548 5728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 12:35:44.133211 5742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:44.134541 5728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:35:44.134663 5728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "e53017687a184a2a94e18aeb3c7d5728"
format_stamp: "Formatted at 2026-06-19 12:35:44 on dist-test-slave-kdrb"
server_key: "4531d15a6b400a07b432712b78754a21"
server_key_iv: "238dea30613f63d29efcf6335bb5e375"
server_key_version: "encryptionkey@0"
I20260619 12:35:44.134769 5728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:44.160566 5728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:44.161301 5728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:44.161463 5728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:44.168618 5728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38601
I20260619 12:35:44.168641 5794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38601 every 8 connection(s)
I20260619 12:35:44.169726 5728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:44.173107 5795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:44.179045 5795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728: Bootstrap starting.
I20260619 12:35:44.179365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 5728
I20260619 12:35:44.179461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:44.179833 392 external_mini_cluster.cc:1428] Setting key 6f1bfb70416a202d9e185b01525f600b
I20260619 12:35:44.181924 5795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:44.182718 5795 log.cc:824] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:44.184947 5795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728: No bootstrap required, opened a new log
I20260619 12:35:44.189484 5795 raft_consensus.cc:348] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53017687a184a2a94e18aeb3c7d5728" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38601 } }
I20260619 12:35:44.189775 5795 raft_consensus.cc:374] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:44.189860 5795 raft_consensus.cc:729] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53017687a184a2a94e18aeb3c7d5728, State: Initialized, Role: FOLLOWER
I20260619 12:35:44.190181 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:44.181310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42886 (local address 127.0.98.62:38601)
0619 12:35:44.181985 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:44.181999 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:44.182036 (+ 37us) server_negotiation.cc:407] Connection header received
0619 12:35:44.182753 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:44.182773 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:44.183343 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:44.183746 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:44.184122 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:44.185176 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:44.186413 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:44.186930 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:44.187059 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:44.188020 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:44.188025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:44.188042 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:35:44.188485 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:44.188547 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:35:44.188568 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:44.188650 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:44.188840 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 12:35:44.189181 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":148,"threads_started":1}
I20260619 12:35:44.190441 5795 consensus_queue.cc:260] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53017687a184a2a94e18aeb3c7d5728" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38601 } }
I20260619 12:35:44.190609 5795 raft_consensus.cc:388] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:44.190675 5795 raft_consensus.cc:482] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:44.190770 5795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:44.191913 5795 raft_consensus.cc:504] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53017687a184a2a94e18aeb3c7d5728" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38601 } }
I20260619 12:35:44.192303 5795 leader_election.cc:302] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e53017687a184a2a94e18aeb3c7d5728; no voters:
I20260619 12:35:44.192710 5795 leader_election.cc:288] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:44.192840 5800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:44.193060 5800 raft_consensus.cc:686] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [term 1 LEADER]: Becoming Leader. State: Replica: e53017687a184a2a94e18aeb3c7d5728, State: Running, Role: LEADER
I20260619 12:35:44.193406 5800 consensus_queue.cc:237] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53017687a184a2a94e18aeb3c7d5728" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38601 } }
I20260619 12:35:44.193480 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:35:44.194090 5795 sys_catalog.cc:564] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:44.195139 5802 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e53017687a184a2a94e18aeb3c7d5728. Latest consensus state: current_term: 1 leader_uuid: "e53017687a184a2a94e18aeb3c7d5728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53017687a184a2a94e18aeb3c7d5728" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38601 } } }
I20260619 12:35:44.195289 5802 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:44.195593 5801 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e53017687a184a2a94e18aeb3c7d5728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53017687a184a2a94e18aeb3c7d5728" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38601 } } }
I20260619 12:35:44.195703 5801 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728 [sys.catalog]: This master's current role is: LEADER
W20260619 12:35:44.199105 5815 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:35:44.199198 5815 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:35:44.199258 5809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:44.200239 5809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:44.205742 5809 catalog_manager.cc:1347] Generated new cluster ID: d9a2b8cb35974a23a5953c8bb8f2db8d
I20260619 12:35:44.205827 5809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:44.214869 5809 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:44.216192 5809 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:44.226042 5809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e53017687a184a2a94e18aeb3c7d5728: Generated new TSK 0
I20260619 12:35:44.226907 5809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:35:44.249773 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:44.244408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42890 (local address 127.0.98.62:38601)
0619 12:35:44.244556 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:44.244561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:44.244753 (+ 192us) server_negotiation.cc:407] Connection header received
0619 12:35:44.244894 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:44.244898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:44.244957 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:44.245087 (+ 130us) server_negotiation.cc:226] Negotiated authn=JWT
0619 12:35:44.245286 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:44.246129 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:44.246988 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:44.247346 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:44.247466 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:44.247815 (+ 349us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 12:35:44.249357 (+ 1542us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33359/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 12:35:44.249613 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42890: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33359/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 12:35:44.249852 5798 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42890: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33359/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 12:35:44.251552 5824 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:35:44.244294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38601 (local address 127.0.0.1:42890)
0619 12:35:44.244646 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:44.244662 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:44.244774 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:44.245088 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:44.245090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:44.245096 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 12:35:44.245186 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:44.245191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:44.246241 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:44.246243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:44.246852 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:44.246859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:44.247459 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:44.247528 (+ 69us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:44.247610 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:44.247636 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 12:35:44.249542 (+ 1906us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33359/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 12:35:44.251343 (+ 1801us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38601: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33359/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":1506,"thread_start_us":162,"threads_started":1}
I20260619 12:35:44.252305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 5728
E20260619 12:35:44.260636 5825 webserver.cc:533] Webserver: sslize failed:
E20260619 12:35:44.260713 5825 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T12:35:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (319 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T12:35:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:44Z Disabled control of system clock
I20260619 12:35:44.299855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45091
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45091
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/testchainca.cert with env {}
W20260619 12:35:44.411003 5829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:44.411286 5829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:44.411326 5829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:44.415558 5829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:44.415620 5829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:44.415637 5829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:44.415653 5829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:44.415709 5829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 12:35:44.415732 5829 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 12:35:44.415769 5829 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 12:35:44.421407 5829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:44.422547 5829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:44.423648 5829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:44.432603 5837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:44.432603 5835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:44.432597 5834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:44.432958 5829 server_base.cc:1034] running on GCE node
I20260619 12:35:44.433617 5829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:44.434855 5829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:44.436074 5829 hybrid_clock.cc:648] HybridClock initialized: now 1781872544436034 us; error 50 us; skew 500 ppm
I20260619 12:35:44.438227 5829 webserver.cc:460] Webserver started at http://127.0.98.62:38015/ using document root <none> and password file <none>
I20260619 12:35:44.438920 5829 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:44.439020 5829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:44.439248 5829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:44.441484 5829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "c51f4ac6aa924efbb2c0d23643a76944"
format_stamp: "Formatted at 2026-06-19 12:35:44 on dist-test-slave-kdrb"
server_key: "7978988c96c036c41a3173f77d01aef7"
server_key_iv: "b965a32f1d7394f56b16cf7bde5efe65"
server_key_version: "encryptionkey@0"
I20260619 12:35:44.442078 5829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "c51f4ac6aa924efbb2c0d23643a76944"
format_stamp: "Formatted at 2026-06-19 12:35:44 on dist-test-slave-kdrb"
server_key: "7978988c96c036c41a3173f77d01aef7"
server_key_iv: "b965a32f1d7394f56b16cf7bde5efe65"
server_key_version: "encryptionkey@0"
I20260619 12:35:44.446128 5829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:35:44.448695 5843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:44.449952 5829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:35:44.450104 5829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "c51f4ac6aa924efbb2c0d23643a76944"
format_stamp: "Formatted at 2026-06-19 12:35:44 on dist-test-slave-kdrb"
server_key: "7978988c96c036c41a3173f77d01aef7"
server_key_iv: "b965a32f1d7394f56b16cf7bde5efe65"
server_key_version: "encryptionkey@0"
I20260619 12:35:44.450233 5829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:44.463779 5829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:44.464457 5829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:44.464659 5829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:44.472038 5829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45091
I20260619 12:35:44.472054 5895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45091 every 8 connection(s)
I20260619 12:35:44.473028 5829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:44.475677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 5829
I20260619 12:35:44.475795 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:44.476076 392 external_mini_cluster.cc:1428] Setting key 5352b2a6bcea1cee301b59dd572b84dd
I20260619 12:35:44.476352 5896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:44.482658 5896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944: Bootstrap starting.
I20260619 12:35:44.485199 5896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:44.485427 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:44.477369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55440 (local address 127.0.98.62:45091)
0619 12:35:44.478044 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:44.478067 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:44.478116 (+ 49us) server_negotiation.cc:407] Connection header received
0619 12:35:44.478922 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:44.478942 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:44.479357 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:44.479767 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:44.480208 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:44.481166 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:44.482064 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:44.482553 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:44.482732 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:44.483641 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:44.483646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:44.483658 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:35:44.484054 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:44.484110 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:35:44.484129 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:44.484243 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:44.484397 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 12:35:44.484658 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":135,"threads_started":1}
I20260619 12:35:44.486013 5896 log.cc:824] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:44.488168 5896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944: No bootstrap required, opened a new log
I20260619 12:35:44.488386 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:35:44.491997 5896 raft_consensus.cc:348] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51f4ac6aa924efbb2c0d23643a76944" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } }
I20260619 12:35:44.492259 5896 raft_consensus.cc:374] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:44.492328 5896 raft_consensus.cc:729] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c51f4ac6aa924efbb2c0d23643a76944, State: Initialized, Role: FOLLOWER
I20260619 12:35:44.492857 5896 consensus_queue.cc:260] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51f4ac6aa924efbb2c0d23643a76944" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } }
I20260619 12:35:44.493031 5896 raft_consensus.cc:388] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:44.493100 5896 raft_consensus.cc:482] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:44.493198 5896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:44.494143 5896 raft_consensus.cc:504] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51f4ac6aa924efbb2c0d23643a76944" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } }
I20260619 12:35:44.494498 5896 leader_election.cc:302] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c51f4ac6aa924efbb2c0d23643a76944; no voters:
I20260619 12:35:44.494837 5896 leader_election.cc:288] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:44.495034 5901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:44.495350 5901 raft_consensus.cc:686] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [term 1 LEADER]: Becoming Leader. State: Replica: c51f4ac6aa924efbb2c0d23643a76944, State: Running, Role: LEADER
I20260619 12:35:44.495708 5901 consensus_queue.cc:237] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51f4ac6aa924efbb2c0d23643a76944" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } }
I20260619 12:35:44.496196 5896 sys_catalog.cc:564] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:44.497469 5902 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c51f4ac6aa924efbb2c0d23643a76944" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51f4ac6aa924efbb2c0d23643a76944" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } } }
I20260619 12:35:44.497610 5902 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:44.497985 5903 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c51f4ac6aa924efbb2c0d23643a76944. Latest consensus state: current_term: 1 leader_uuid: "c51f4ac6aa924efbb2c0d23643a76944" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51f4ac6aa924efbb2c0d23643a76944" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } } }
I20260619 12:35:44.498090 5903 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:44.498056 5910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:44.501027 5910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:44.506932 5910 catalog_manager.cc:1347] Generated new cluster ID: 4e17b245389e430d880ddd6b74522ba8
I20260619 12:35:44.507047 5910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:44.514954 5910 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:44.516254 5910 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:44.522049 5910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c51f4ac6aa924efbb2c0d23643a76944: Generated new TSK 0
I20260619 12:35:44.522786 5910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:35:44.699782 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:44.535585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.0.98.62:45091)
0619 12:35:44.535755 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:44.535760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:44.535975 (+ 215us) server_negotiation.cc:407] Connection header received
0619 12:35:44.536145 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:44.536148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:44.536201 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:44.536303 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0619 12:35:44.536554 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:44.537092 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:44.538364 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:44.538582 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:44.538667 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:44.539004 (+ 337us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 12:35:44.699320 (+160316us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 12:35:44.699658 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55446: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 12:35:44.699890 5899 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55446: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 12:35:44.699972 5925 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:35:44.535479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45091 (local address 127.0.0.1:55446)
0619 12:35:44.535860 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:44.535876 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:44.536013 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:44.536324 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:44.536327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:44.536335 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 12:35:44.536452 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:44.536458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:44.537222 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:44.537226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:44.538233 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:44.538244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:44.538681 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:44.538707 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:44.538806 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:44.538836 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 12:35:44.699586 (+160750us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 12:35:44.699769 (+ 183us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45091: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":136,"threads_started":1}
I20260619 12:35:44.700575 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 5829
2026-06-19T12:35:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (428 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (126 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (126 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:35:45 dist-test-slave-kdrb krb5kdc[5937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:35:45 dist-test-slave-kdrb krb5kdc[5937](info): set up 2 sockets
Jun 19 12:35:45 dist-test-slave-kdrb krb5kdc[5937](info): commencing operation
krb5kdc: starting...
W20260619 12:35:46.013705 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 12:35:46.273355 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.260s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 12:35:46.504560 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.231s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260619 12:35:46.609498 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.105s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/test-user.keytab.
W20260619 12:35:46.998163 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.389s user 0.001s sys 0.001s
Jun 19 12:35:47 dist-test-slave-kdrb krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:35:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:35:47.046638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39559
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:47.160104 5950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:47.160458 5950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:47.160542 5950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:47.164788 5950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:47.164909 5950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:47.164947 5950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:47.165002 5950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:47.165043 5950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:47.170642 5950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:47.172000 5950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:47.173122 5950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:47.182063 5956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:47.182071 5955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:47.182163 5950 server_base.cc:1034] running on GCE node
W20260619 12:35:47.182071 5958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:47.183004 5950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:47.184264 5950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:47.185477 5950 hybrid_clock.cc:648] HybridClock initialized: now 1781872547185441 us; error 52 us; skew 500 ppm
Jun 19 12:35:47 dist-test-slave-kdrb krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:47.193161 5950 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:35:47.194176 5950 webserver.cc:460] Webserver started at http://127.0.98.62:43655/ using document root <none> and password file <none>
I20260619 12:35:47.194769 5950 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:47.194860 5950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:47.195092 5950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:47.197288 5950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "86420e4b08054d06ad6b0c1cb61621a9"
format_stamp: "Formatted at 2026-06-19 12:35:47 on dist-test-slave-kdrb"
server_key: "f20ece38356e0a6a8dc5042d7e7b48ba"
server_key_iv: "5b90a7b71ec887ce75c17dded96ca1e3"
server_key_version: "encryptionkey@0"
I20260619 12:35:47.197860 5950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "86420e4b08054d06ad6b0c1cb61621a9"
format_stamp: "Formatted at 2026-06-19 12:35:47 on dist-test-slave-kdrb"
server_key: "f20ece38356e0a6a8dc5042d7e7b48ba"
server_key_iv: "5b90a7b71ec887ce75c17dded96ca1e3"
server_key_version: "encryptionkey@0"
I20260619 12:35:47.201607 5950 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 12:35:47.204115 5965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:47.205366 5950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:35:47.205508 5950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "86420e4b08054d06ad6b0c1cb61621a9"
format_stamp: "Formatted at 2026-06-19 12:35:47 on dist-test-slave-kdrb"
server_key: "f20ece38356e0a6a8dc5042d7e7b48ba"
server_key_iv: "5b90a7b71ec887ce75c17dded96ca1e3"
server_key_version: "encryptionkey@0"
I20260619 12:35:47.205628 5950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:47.236529 5950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:47.239647 5950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:47.239856 5950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:47.246989 5950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39559
I20260619 12:35:47.248271 5950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:47.249372 6017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39559 every 8 connection(s)
I20260619 12:35:47.252584 6018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:47.253386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 5950
I20260619 12:35:47.253528 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:47.253789 392 external_mini_cluster.cc:1428] Setting key d824e4121f442040a7ef2e0754516290
I20260619 12:35:47.260957 6018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9: Bootstrap starting.
Jun 19 12:35:47 dist-test-slave-kdrb krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:47.263363 6018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:47.264220 6018 log.cc:824] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:47.266155 6018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9: No bootstrap required, opened a new log
I20260619 12:35:47.268679 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:47.255074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.0.98.62:39559)
0619 12:35:47.255510 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:47.255530 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:47.256111 (+ 581us) server_negotiation.cc:407] Connection header received
0619 12:35:47.256906 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:47.256924 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:47.257269 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:47.257679 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:47.258512 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:47.259556 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:47.260406 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:47.260884 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:47.261053 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:47.263527 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:47.263538 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:47.263553 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:47.263581 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:47.265434 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:47.265917 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:47.265922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:47.265926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:47.266039 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:47.266352 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:47.266356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:47.266357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:47.266867 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:47.267086 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:47.267289 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 12:35:47.267629 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":138,"threads_started":1}
I20260619 12:35:47.271240 6018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86420e4b08054d06ad6b0c1cb61621a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39559 } }
I20260619 12:35:47.271562 6018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:47.271610 6018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86420e4b08054d06ad6b0c1cb61621a9, State: Initialized, Role: FOLLOWER
I20260619 12:35:47.272004 6018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86420e4b08054d06ad6b0c1cb61621a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39559 } }
I20260619 12:35:47.272128 6018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:47.272168 6018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:47.272238 6018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:47.273155 6018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86420e4b08054d06ad6b0c1cb61621a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39559 } }
I20260619 12:35:47.273452 6018 leader_election.cc:302] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86420e4b08054d06ad6b0c1cb61621a9; no voters:
I20260619 12:35:47.273716 6018 leader_election.cc:288] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:47.273846 6023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:47.274096 6023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [term 1 LEADER]: Becoming Leader. State: Replica: 86420e4b08054d06ad6b0c1cb61621a9, State: Running, Role: LEADER
I20260619 12:35:47.274434 6023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86420e4b08054d06ad6b0c1cb61621a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39559 } }
I20260619 12:35:47.274909 6018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:47.276093 6025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86420e4b08054d06ad6b0c1cb61621a9. Latest consensus state: current_term: 1 leader_uuid: "86420e4b08054d06ad6b0c1cb61621a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86420e4b08054d06ad6b0c1cb61621a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39559 } } }
I20260619 12:35:47.276226 6025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:47.276605 6032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:47.277549 6024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86420e4b08054d06ad6b0c1cb61621a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86420e4b08054d06ad6b0c1cb61621a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39559 } } }
I20260619 12:35:47.277647 6024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:47.279923 6032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:47.285820 6032 catalog_manager.cc:1347] Generated new cluster ID: b0bc9546284947f0af081d7e6eec5ca7
I20260619 12:35:47.285913 6032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:47.298184 6032 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:47.299516 6032 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:47.306588 6032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86420e4b08054d06ad6b0c1cb61621a9: Generated new TSK 0
I20260619 12:35:47.307405 6032 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:35:47.340317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39559
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:47.452054 6044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:47.452380 6044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:47.452474 6044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:47.456571 6044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:47.456670 6044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:47.456764 6044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:35:47.462124 6044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:47.463245 6044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:47.464336 6044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:47.474465 6049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:47.474465 6050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:47.474470 6052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:47.474539 6044 server_base.cc:1034] running on GCE node
I20260619 12:35:47.475333 6044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:47.476094 6044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:47.477322 6044 hybrid_clock.cc:648] HybridClock initialized: now 1781872547477300 us; error 40 us; skew 500 ppm
Jun 19 12:35:47 dist-test-slave-kdrb krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:47.480345 6044 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:35:47.481331 6044 webserver.cc:460] Webserver started at http://127.0.98.1:41147/ using document root <none> and password file <none>
I20260619 12:35:47.481943 6044 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:47.482038 6044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:47.482270 6044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:47.484469 6044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "bd064a0382c249cdaf65e7a92cd277d2"
format_stamp: "Formatted at 2026-06-19 12:35:47 on dist-test-slave-kdrb"
server_key: "b775e0d77346426abb91a82ee445265c"
server_key_iv: "d0ba292948b1f33ba397aa1da9fce9db"
server_key_version: "encryptionkey@0"
I20260619 12:35:47.485059 6044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bd064a0382c249cdaf65e7a92cd277d2"
format_stamp: "Formatted at 2026-06-19 12:35:47 on dist-test-slave-kdrb"
server_key: "b775e0d77346426abb91a82ee445265c"
server_key_iv: "d0ba292948b1f33ba397aa1da9fce9db"
server_key_version: "encryptionkey@0"
I20260619 12:35:47.488909 6044 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:35:47.491569 6059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:47.492825 6044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 12:35:47.492960 6044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "bd064a0382c249cdaf65e7a92cd277d2"
format_stamp: "Formatted at 2026-06-19 12:35:47 on dist-test-slave-kdrb"
server_key: "b775e0d77346426abb91a82ee445265c"
server_key_iv: "d0ba292948b1f33ba397aa1da9fce9db"
server_key_version: "encryptionkey@0"
I20260619 12:35:47.493075 6044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:47.503654 6044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:47.507011 6044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:47.507215 6044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:47.507862 6044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:47.508685 6044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:47.508731 6044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:47.508816 6044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:47.508867 6044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:47.517819 6044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37701
I20260619 12:35:47.517859 6172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37701 every 8 connection(s)
I20260619 12:35:47.518786 6044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:35:47.527088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 6044
I20260619 12:35:47.527240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:35:47.527642 392 external_mini_cluster.cc:1428] Setting key 9d5fcafd596c684091bb8204ce6f0c76
Jun 19 12:35:47 dist-test-slave-kdrb krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:47.533442 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:47.520686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39913 (local address 127.0.98.62:39559)
0619 12:35:47.520806 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:47.520810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:47.521689 (+ 879us) server_negotiation.cc:407] Connection header received
0619 12:35:47.523123 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:47.523126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:47.523185 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:47.523289 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:47.524599 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:47.525144 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:47.526338 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:47.526578 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:47.526689 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:47.529447 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:47.529450 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:47.529452 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:47.529476 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:47.531006 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:47.531712 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:47.531715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:47.531716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:47.531792 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:47.532211 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:47.532213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:47.532214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:47.532395 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:47.532477 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:47.533189 (+ 712us) server_negotiation.cc:299] Negotiation successful
0619 12:35:47.533304 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 12:35:47.534854 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:47.521073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39559 (local address 127.0.98.1:39913)
0619 12:35:47.521537 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:47.521582 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:47.522896 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:47.523533 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:47.523543 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:47.523935 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:47.524440 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:47.524454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:47.525276 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:47.525279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:47.526210 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:47.526219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:47.526747 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:47.526751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:47.526945 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:47.527647 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:47.527665 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:47.529264 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:47.531134 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:47.531141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:47.531165 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:47.531609 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:47.531880 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:47.531886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:47.531891 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:47.532114 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:47.532503 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:47.532514 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:47.532884 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 12:35:47.533143 (+ 259us) client_negotiation.cc:240] Negotiation successful
0619 12:35:47.533557 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":116,"threads_started":1}
I20260619 12:35:47.536382 6173 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39559
I20260619 12:35:47.536657 6173 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:47.537182 6173 heartbeater.cc:511] Master 127.0.98.62:39559 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:35:47.538763 5982 ts_manager.cc:194] Registered new tserver with Master: bd064a0382c249cdaf65e7a92cd277d2 (127.0.98.1:37701)
I20260619 12:35:47.540019 5982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:35:47.556964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39559
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:47.668152 6178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:47.668444 6178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:47.668489 6178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:47.673291 6178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:47.673364 6178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:47.673468 6178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:35:47.679276 6178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:47.680455 6178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:47.681519 6178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:47.691340 6183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:47.691344 6184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:47.691516 6178 server_base.cc:1034] running on GCE node
W20260619 12:35:47.691344 6186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:47.692196 6178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:47.692884 6178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:47.694063 6178 hybrid_clock.cc:648] HybridClock initialized: now 1781872547694033 us; error 45 us; skew 500 ppm
Jun 19 12:35:47 dist-test-slave-kdrb krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:47.697026 6178 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:35:47.697948 6178 webserver.cc:460] Webserver started at http://127.0.98.2:39935/ using document root <none> and password file <none>
I20260619 12:35:47.698479 6178 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:47.698536 6178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:47.698699 6178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:47.700744 6178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "b84b5dbe791c48799f18f471ba5a75af"
format_stamp: "Formatted at 2026-06-19 12:35:47 on dist-test-slave-kdrb"
server_key: "3e273740cb245ffc0e3c1a77fb166756"
server_key_iv: "2c05fe563e1dea683ed292ad6903fe12"
server_key_version: "encryptionkey@0"
I20260619 12:35:47.701258 6178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b84b5dbe791c48799f18f471ba5a75af"
format_stamp: "Formatted at 2026-06-19 12:35:47 on dist-test-slave-kdrb"
server_key: "3e273740cb245ffc0e3c1a77fb166756"
server_key_iv: "2c05fe563e1dea683ed292ad6903fe12"
server_key_version: "encryptionkey@0"
I20260619 12:35:47.704991 6178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:35:47.707314 6193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:47.708595 6178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:35:47.708727 6178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "b84b5dbe791c48799f18f471ba5a75af"
format_stamp: "Formatted at 2026-06-19 12:35:47 on dist-test-slave-kdrb"
server_key: "3e273740cb245ffc0e3c1a77fb166756"
server_key_iv: "2c05fe563e1dea683ed292ad6903fe12"
server_key_version: "encryptionkey@0"
I20260619 12:35:47.708839 6178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:47.725044 6178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:47.728482 6178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:47.728693 6178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:47.729321 6178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:47.730151 6178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:47.730218 6178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:47.730291 6178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:47.730333 6178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:47.740110 6178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44275
I20260619 12:35:47.740151 6306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44275 every 8 connection(s)
I20260619 12:35:47.741045 6178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:35:47.743461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 6178
I20260619 12:35:47.743651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:35:47.743985 392 external_mini_cluster.cc:1428] Setting key 140d1d6ae10e75d62416305dd13c4d7c
Jun 19 12:35:47 dist-test-slave-kdrb krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:35:47.755892 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:47.743044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45393 (local address 127.0.98.62:39559)
0619 12:35:47.743210 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:47.743215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:47.744321 (+ 1106us) server_negotiation.cc:407] Connection header received
0619 12:35:47.745965 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:47.745968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:47.746023 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:47.746118 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:47.747362 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:47.747983 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:47.749161 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:47.749406 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:47.749516 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:47.752068 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:47.752072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:47.752075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:47.752109 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:47.753621 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:47.754188 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:47.754191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:47.754192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:47.754272 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:47.754576 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:47.754578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:47.754580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:47.754763 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:47.754842 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:47.755441 (+ 599us) server_negotiation.cc:299] Negotiation successful
0619 12:35:47.755635 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:35:47.756687 6309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:47.743549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39559 (local address 127.0.98.2:45393)
0619 12:35:47.744129 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:47.744187 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:47.745736 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:47.746296 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:47.746303 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:47.746676 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:47.747203 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:47.747217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:47.748114 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:47.748118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:47.749030 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:47.749039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:47.749562 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:47.749566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:47.749751 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:47.750385 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:47.750402 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:47.751888 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:47.753738 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:47.753744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:47.753762 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:47.754089 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:47.754357 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:47.754359 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:47.754360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:47.754496 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:47.754861 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:47.754866 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:47.755219 (+ 353us) client_negotiation.cc:769] Sending connection context
0619 12:35:47.755397 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 12:35:47.755814 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":145,"threads_started":1}
I20260619 12:35:47.757889 6307 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39559
I20260619 12:35:47.758177 6307 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:47.758764 6307 heartbeater.cc:511] Master 127.0.98.62:39559 requested a full tablet report, sending...
I20260619 12:35:47.759945 5982 ts_manager.cc:194] Registered new tserver with Master: b84b5dbe791c48799f18f471ba5a75af (127.0.98.2:44275)
I20260619 12:35:47.760519 5982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:35:47.772209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39559
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:47.885111 6312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:47.885358 6312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:47.885399 6312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:47.889581 6312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:47.889645 6312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:47.889704 6312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:35:47.895263 6312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:47.896319 6312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:47.897385 6312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:47.906785 6318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:47.906812 6320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:47.906903 6312 server_base.cc:1034] running on GCE node
W20260619 12:35:47.906786 6317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:47.907436 6312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:47.908092 6312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:47.909252 6312 hybrid_clock.cc:648] HybridClock initialized: now 1781872547909239 us; error 32 us; skew 500 ppm
Jun 19 12:35:47 dist-test-slave-kdrb krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:47.912166 6312 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:35:47.913163 6312 webserver.cc:460] Webserver started at http://127.0.98.3:42369/ using document root <none> and password file <none>
I20260619 12:35:47.913730 6312 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:47.913817 6312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:47.914047 6312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:47.916188 6312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "cd102914626a4e3ba8f2b49e38912101"
format_stamp: "Formatted at 2026-06-19 12:35:47 on dist-test-slave-kdrb"
server_key: "6fa2012324d1dd4fd30fc3d4bd97bbfb"
server_key_iv: "974094b83ccf27218f5bdeeb7048dc12"
server_key_version: "encryptionkey@0"
I20260619 12:35:47.916772 6312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cd102914626a4e3ba8f2b49e38912101"
format_stamp: "Formatted at 2026-06-19 12:35:47 on dist-test-slave-kdrb"
server_key: "6fa2012324d1dd4fd30fc3d4bd97bbfb"
server_key_iv: "974094b83ccf27218f5bdeeb7048dc12"
server_key_version: "encryptionkey@0"
I20260619 12:35:47.920842 6312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:35:47.923607 6327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:47.924866 6312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:35:47.925016 6312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "cd102914626a4e3ba8f2b49e38912101"
format_stamp: "Formatted at 2026-06-19 12:35:47 on dist-test-slave-kdrb"
server_key: "6fa2012324d1dd4fd30fc3d4bd97bbfb"
server_key_iv: "974094b83ccf27218f5bdeeb7048dc12"
server_key_version: "encryptionkey@0"
I20260619 12:35:47.925138 6312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:47.948065 6312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:47.951630 6312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:47.951843 6312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:47.952478 6312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:47.953317 6312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:47.953387 6312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:47.953454 6312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:47.953504 6312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:47.962258 6312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33717
I20260619 12:35:47.962278 6440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33717 every 8 connection(s)
I20260619 12:35:47.963174 6312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:35:47.969136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 6312
I20260619 12:35:47.969264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:35:47.969552 392 external_mini_cluster.cc:1428] Setting key 45882b090efbf765f925e9fe97bd91d1
Jun 19 12:35:47 dist-test-slave-kdrb krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:47.977560 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:47.965047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59235 (local address 127.0.98.62:39559)
0619 12:35:47.965182 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:47.965186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:47.966040 (+ 854us) server_negotiation.cc:407] Connection header received
0619 12:35:47.967431 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:47.967435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:47.967525 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:47.967628 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:47.968778 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:47.969587 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:47.971143 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:47.971386 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:47.971509 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:47.973960 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:47.973964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:47.973967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:47.973998 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:47.975542 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:47.976036 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:47.976041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:47.976043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:47.976120 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:47.976518 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:47.976523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:47.976525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:47.976701 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:47.976805 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:47.977285 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 12:35:47.977402 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:35:47.978370 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:47.965445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39559 (local address 127.0.98.3:59235)
0619 12:35:47.965894 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:47.965936 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:47.967202 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:47.967804 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:47.967811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:47.968165 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:47.968638 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:47.968650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:47.969721 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:47.969725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:47.971029 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:47.971040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:47.971560 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:47.971564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:47.971744 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:47.972308 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:47.972324 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:47.973795 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:47.975649 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:47.975655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:47.975669 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:47.975933 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:47.976208 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:47.976212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:47.976214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:47.976427 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:47.976803 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:47.976810 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:47.977052 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 12:35:47.977234 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 12:35:47.977562 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":114,"threads_started":1}
I20260619 12:35:47.979419 6441 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39559
I20260619 12:35:47.979687 6441 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:47.980151 6441 heartbeater.cc:511] Master 127.0.98.62:39559 requested a full tablet report, sending...
I20260619 12:35:47.981120 5982 ts_manager.cc:194] Registered new tserver with Master: cd102914626a4e3ba8f2b49e38912101 (127.0.98.3:33717)
I20260619 12:35:47.981675 5982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59235
I20260619 12:35:47.983207 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/test-admin.keytab.
Jun 19 12:35:47 dist-test-slave-kdrb krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:47.999835 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 12:35:48.002661 6448 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:35:48.001138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.0.98.1:37701)
0619 12:35:48.001482 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:48.001502 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:48.001531 (+ 29us) server_negotiation.cc:407] Connection header received
0619 12:35:48.001591 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:48.001596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:48.001782 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:48.001954 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:48.002467 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51060: BlockingRecv error: recv got EOF from 127.0.0.1:51060 (error 108)
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":117,"threads_started":1}
W20260619 12:35:48.002753 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 12:35:47 dist-test-slave-kdrb krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:48.003811 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 12:35:47 dist-test-slave-kdrb krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:48.004709 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 12:35:48 dist-test-slave-kdrb krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:35:48.014431 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:48.005594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.0.98.1:37701)
0619 12:35:48.005719 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:48.005724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:48.005824 (+ 100us) server_negotiation.cc:407] Connection header received
0619 12:35:48.005993 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:48.005995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:48.006034 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:48.006113 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:48.006635 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:48.007407 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:48.008248 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:48.008636 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:48.008748 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:48.011013 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:48.011023 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:48.011042 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:48.011064 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:48.012634 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:48.013140 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:48.013150 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:48.013152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:48.013226 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:48.013480 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:48.013485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:48.013487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:48.013729 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:48.013940 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:48.014115 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 12:35:48.014263 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 12:35:48.017951 6448 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:35:48.017111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51082 (local address 127.0.98.1:37701)
0619 12:35:48.017260 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:48.017263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:48.017311 (+ 48us) server_negotiation.cc:407] Connection header received
0619 12:35:48.017406 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:48.017408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:48.017442 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:48.017500 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:48.017827 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51082: BlockingRecv error: recv got EOF from 127.0.0.1:51082 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 12:35:48.018024 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 12:35:48 dist-test-slave-kdrb krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:48.018956 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 12:35:48 dist-test-slave-kdrb krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:48.019799 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 12:35:48 dist-test-slave-kdrb krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:35:48.027932 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:48.020558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.0.98.1:37701)
0619 12:35:48.020650 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:48.020653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:48.020822 (+ 169us) server_negotiation.cc:407] Connection header received
0619 12:35:48.020898 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:48.020901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:48.020935 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:48.020997 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:48.021386 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:48.021912 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:48.022625 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:48.022837 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:48.022923 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:48.024948 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:48.024951 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:48.024953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:48.024974 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:48.026355 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:48.026792 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:48.026796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:48.026797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:48.026876 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:48.027094 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:48.027096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:48.027097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:48.027266 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:48.027361 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:48.027606 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 12:35:48.027719 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260619 12:35:48.030680 6448 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:35:48.029708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51100 (local address 127.0.98.1:37701)
0619 12:35:48.029832 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:48.029836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:48.029941 (+ 105us) server_negotiation.cc:407] Connection header received
0619 12:35:48.030077 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:48.030080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:48.030117 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:48.030196 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:48.030563 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51100: BlockingRecv error: recv got EOF from 127.0.0.1:51100 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 12:35:48.031056 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872516972220-392-0/krb5kdc/krb5cc)
Jun 19 12:35:48 dist-test-slave-kdrb krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:48.032392 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 12:35:48 dist-test-slave-kdrb krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:48.033398 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 12:35:48 dist-test-slave-kdrb krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:35:48.041954 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:48.034267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.0.98.1:37701)
0619 12:35:48.034397 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:48.034401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:48.034514 (+ 113us) server_negotiation.cc:407] Connection header received
0619 12:35:48.034677 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:48.034679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:48.034715 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:48.034800 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:48.035306 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:48.035890 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:48.036674 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:48.036898 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:48.037004 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:48.039197 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:48.039200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:48.039202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:48.039223 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:48.040442 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:48.040834 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:48.040837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:48.040838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:48.040916 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:48.041165 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:48.041168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:48.041169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:48.041341 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:48.041451 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:48.041633 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 12:35:48.041766 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:35:48.043025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 6044
I20260619 12:35:48.048772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 6178
I20260619 12:35:48.054173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 6312
I20260619 12:35:48.059237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 5950
2026-06-19T12:35:48Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3107 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:35:48 dist-test-slave-kdrb krb5kdc[6461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:35:48 dist-test-slave-kdrb krb5kdc[6461](info): set up 2 sockets
Jun 19 12:35:48 dist-test-slave-kdrb krb5kdc[6461](info): commencing operation
krb5kdc: starting...
W20260619 12:35:49.111941 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:35:49 dist-test-slave-kdrb krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:35:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 12:35:49.242242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43151
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:49.360500 6474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:49.360805 6474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:49.360888 6474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:49.365015 6474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:49.365092 6474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:49.365115 6474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:49.365136 6474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:49.365195 6474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:49.370770 6474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:49.371922 6474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:49.373016 6474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:49.381708 6482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:49.381649 6480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:49.381687 6474 server_base.cc:1034] running on GCE node
W20260619 12:35:49.381656 6479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:49.382498 6474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:49.383703 6474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:49.384871 6474 hybrid_clock.cc:648] HybridClock initialized: now 1781872549384854 us; error 33 us; skew 500 ppm
Jun 19 12:35:49 dist-test-slave-kdrb krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872549, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:49.387624 6474 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 12:35:49.388453 6474 webserver.cc:460] Webserver started at http://127.0.98.62:36015/ using document root <none> and password file <none>
I20260619 12:35:49.388957 6474 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:49.389015 6474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:49.389175 6474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:49.391110 6474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "5a6c5cff1ee44943a5f87b30f25d9083"
format_stamp: "Formatted at 2026-06-19 12:35:49 on dist-test-slave-kdrb"
server_key: "1ab62a0c265d918e65fd10f70d905f38"
server_key_iv: "a139978d5cf88f23093b051ed51b7c62"
server_key_version: "encryptionkey@0"
I20260619 12:35:49.391686 6474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "5a6c5cff1ee44943a5f87b30f25d9083"
format_stamp: "Formatted at 2026-06-19 12:35:49 on dist-test-slave-kdrb"
server_key: "1ab62a0c265d918e65fd10f70d905f38"
server_key_iv: "a139978d5cf88f23093b051ed51b7c62"
server_key_version: "encryptionkey@0"
I20260619 12:35:49.395221 6474 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 12:35:49.397629 6489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:49.398895 6474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:35:49.398994 6474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "5a6c5cff1ee44943a5f87b30f25d9083"
format_stamp: "Formatted at 2026-06-19 12:35:49 on dist-test-slave-kdrb"
server_key: "1ab62a0c265d918e65fd10f70d905f38"
server_key_iv: "a139978d5cf88f23093b051ed51b7c62"
server_key_version: "encryptionkey@0"
I20260619 12:35:49.399080 6474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:49.423636 6474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:49.426694 6474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:49.426870 6474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:49.433991 6474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43151
I20260619 12:35:49.434279 6541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43151 every 8 connection(s)
I20260619 12:35:49.435030 6474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:49.438028 6542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:49.439193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 6474
I20260619 12:35:49.439324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:49.439674 392 external_mini_cluster.cc:1428] Setting key 309c00260c77bba44fd73add27ba7512
I20260619 12:35:49.444850 6542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083: Bootstrap starting.
I20260619 12:35:49.447701 6542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:49.448601 6542 log.cc:824] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:35:49 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:35:49.450587 6542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083: No bootstrap required, opened a new log
I20260619 12:35:49.454669 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:49.441065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.0.98.62:43151)
0619 12:35:49.441505 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:49.441528 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:49.441563 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:35:49.442217 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:49.442236 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:49.442590 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:49.442889 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:49.443827 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:49.444809 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:49.445935 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:49.446400 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:49.446514 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:49.449368 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:49.449374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:49.449394 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:49.449422 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:49.451128 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:49.451600 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:49.451606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:49.451612 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:49.451754 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:49.452097 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:49.452100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:49.452102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:49.452658 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:49.452874 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:49.453294 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 12:35:49.453634 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":145,"threads_started":1}
I20260619 12:35:49.455426 6542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } }
I20260619 12:35:49.455773 6542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:49.455884 6542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a6c5cff1ee44943a5f87b30f25d9083, State: Initialized, Role: FOLLOWER
I20260619 12:35:49.456386 6542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } }
I20260619 12:35:49.456559 6542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:49.456629 6542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:49.456743 6542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:49.457799 6542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } }
I20260619 12:35:49.458125 6542 leader_election.cc:302] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a6c5cff1ee44943a5f87b30f25d9083; no voters:
I20260619 12:35:49.458461 6542 leader_election.cc:288] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:49.458609 6547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:49.458832 6547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [term 1 LEADER]: Becoming Leader. State: Replica: 5a6c5cff1ee44943a5f87b30f25d9083, State: Running, Role: LEADER
I20260619 12:35:49.459214 6547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } }
I20260619 12:35:49.459941 6542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:49.460932 6549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a6c5cff1ee44943a5f87b30f25d9083. Latest consensus state: current_term: 1 leader_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } }
I20260619 12:35:49.461139 6549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:49.461190 6548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } }
I20260619 12:35:49.461266 6548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:49.461619 6556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:49.464836 6556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:49.470875 6556 catalog_manager.cc:1347] Generated new cluster ID: 88dd114f19f84679bc91e6d4d7d7c049
I20260619 12:35:49.470963 6556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:49.483014 6556 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:49.484061 6556 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:49.490958 6556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083: Generated new TSK 0
I20260619 12:35:49.491726 6556 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 12:35:49.522810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43151
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:49.635682 6568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:49.635962 6568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:49.636006 6568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:49.640188 6568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:49.640259 6568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:49.640319 6568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:35:49.645597 6568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:49.646629 6568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:49.647675 6568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:49.657316 6574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:49.657521 6568 server_base.cc:1034] running on GCE node
W20260619 12:35:49.657316 6576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:49.657320 6573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:49.658084 6568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:49.658840 6568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:49.659999 6568 hybrid_clock.cc:648] HybridClock initialized: now 1781872549659990 us; error 32 us; skew 500 ppm
Jun 19 12:35:49 dist-test-slave-kdrb krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872549, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:49.662973 6568 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 12:35:49.663865 6568 webserver.cc:460] Webserver started at http://127.0.98.1:41251/ using document root <none> and password file <none>
I20260619 12:35:49.664400 6568 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:49.664460 6568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:49.664688 6568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:49.666702 6568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "114c34026d3c4740a1794ba290c620f5"
format_stamp: "Formatted at 2026-06-19 12:35:49 on dist-test-slave-kdrb"
server_key: "a785c40fb270b989187e64a1d865711e"
server_key_iv: "7698f05eababe07077f93085a9f85bb4"
server_key_version: "encryptionkey@0"
I20260619 12:35:49.667220 6568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "114c34026d3c4740a1794ba290c620f5"
format_stamp: "Formatted at 2026-06-19 12:35:49 on dist-test-slave-kdrb"
server_key: "a785c40fb270b989187e64a1d865711e"
server_key_iv: "7698f05eababe07077f93085a9f85bb4"
server_key_version: "encryptionkey@0"
I20260619 12:35:49.670888 6568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 12:35:49.673324 6583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:49.674567 6568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:35:49.674665 6568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "114c34026d3c4740a1794ba290c620f5"
format_stamp: "Formatted at 2026-06-19 12:35:49 on dist-test-slave-kdrb"
server_key: "a785c40fb270b989187e64a1d865711e"
server_key_iv: "7698f05eababe07077f93085a9f85bb4"
server_key_version: "encryptionkey@0"
I20260619 12:35:49.674757 6568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:49.687906 6568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:49.690816 6568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:49.690986 6568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:49.691568 6568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:49.692363 6568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:49.692412 6568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:49.692466 6568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:49.692481 6568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:49.701855 6568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34107
I20260619 12:35:49.701902 6696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34107 every 8 connection(s)
I20260619 12:35:49.702979 6568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:35:49.709172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 6568
I20260619 12:35:49.709314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:35:49.709612 392 external_mini_cluster.cc:1428] Setting key 8dafee25985a93a332544e8bf24f5b34
Jun 19 12:35:49 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872549, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:35:49.718343 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:49.704946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60309 (local address 127.0.98.62:43151)
0619 12:35:49.705084 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:49.705092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:49.706029 (+ 937us) server_negotiation.cc:407] Connection header received
0619 12:35:49.707795 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:49.707798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:49.707867 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:49.707974 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:49.709194 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:49.709873 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:49.711049 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:49.711279 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:49.711389 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:49.714274 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:49.714281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:49.714287 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:49.714324 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:49.716139 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:49.716688 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:49.716693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:49.716697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:49.716801 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:49.717146 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:49.717150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:49.717151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:49.717398 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:49.717512 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:49.718000 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 12:35:49.718143 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:35:49.719120 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:49.705377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.1:60309)
0619 12:35:49.705870 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:49.705914 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:49.707570 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:49.708160 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:49.708168 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:49.708557 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:49.709047 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:49.709060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:49.710003 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:49.710006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:49.710916 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:49.710925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:49.711431 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:49.711435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:49.711640 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:49.712284 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:49.712301 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:49.714078 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:49.716271 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:49.716276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:49.716291 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:49.716580 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:49.716910 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:49.716912 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:49.716914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:49.717053 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:49.717499 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:49.717504 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:49.717751 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 12:35:49.717913 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 12:35:49.718190 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:35:49.720432 6697 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43151
I20260619 12:35:49.720692 6697 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:49.721211 6697 heartbeater.cc:511] Master 127.0.98.62:43151 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260619 12:35:49.722978 6506 ts_manager.cc:194] Registered new tserver with Master: 114c34026d3c4740a1794ba290c620f5 (127.0.98.1:34107)
I20260619 12:35:49.724457 6506 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 12:35:49.740007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43151
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:49.852222 6702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:49.852517 6702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:49.852561 6702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:49.856818 6702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:49.856885 6702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:49.856945 6702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:35:49.862345 6702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:49.863364 6702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:49.864431 6702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:49.873822 6707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:49.874013 6702 server_base.cc:1034] running on GCE node
W20260619 12:35:49.873864 6710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:49.873905 6708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:49.874581 6702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:49.875211 6702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:49.876366 6702 hybrid_clock.cc:648] HybridClock initialized: now 1781872549876342 us; error 45 us; skew 500 ppm
Jun 19 12:35:49 dist-test-slave-kdrb krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872549, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:49.879163 6702 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 12:35:49.880079 6702 webserver.cc:460] Webserver started at http://127.0.98.2:42101/ using document root <none> and password file <none>
I20260619 12:35:49.880635 6702 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:49.880697 6702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:49.880936 6702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:49.882931 6702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "f5bb8dd6c140400b91cceefd251251c6"
format_stamp: "Formatted at 2026-06-19 12:35:49 on dist-test-slave-kdrb"
server_key: "0387c19c7419ca6646bfb6320cea5a32"
server_key_iv: "6b37b4bee382cbd0985db7639e98d95d"
server_key_version: "encryptionkey@0"
I20260619 12:35:49.883535 6702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f5bb8dd6c140400b91cceefd251251c6"
format_stamp: "Formatted at 2026-06-19 12:35:49 on dist-test-slave-kdrb"
server_key: "0387c19c7419ca6646bfb6320cea5a32"
server_key_iv: "6b37b4bee382cbd0985db7639e98d95d"
server_key_version: "encryptionkey@0"
I20260619 12:35:49.887166 6702 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:35:49.889595 6717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:49.890817 6702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:35:49.890954 6702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "f5bb8dd6c140400b91cceefd251251c6"
format_stamp: "Formatted at 2026-06-19 12:35:49 on dist-test-slave-kdrb"
server_key: "0387c19c7419ca6646bfb6320cea5a32"
server_key_iv: "6b37b4bee382cbd0985db7639e98d95d"
server_key_version: "encryptionkey@0"
I20260619 12:35:49.891064 6702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:49.905231 6702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:49.908362 6702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:49.908560 6702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:49.909144 6702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:49.909998 6702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:49.910066 6702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:49.910133 6702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:49.910179 6702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:49.918860 6702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37503
I20260619 12:35:49.918880 6830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37503 every 8 connection(s)
I20260619 12:35:49.919896 6702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:35:49.926277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 6702
I20260619 12:35:49.926433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:35:49.926743 392 external_mini_cluster.cc:1428] Setting key 29adebb65e33e04c6c959c1826c07018
Jun 19 12:35:49 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872549, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:35:49.935158 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:49.921796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47931 (local address 127.0.98.62:43151)
0619 12:35:49.921933 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:49.921936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:49.922796 (+ 860us) server_negotiation.cc:407] Connection header received
0619 12:35:49.924107 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:49.924110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:49.924159 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:49.924240 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:49.925412 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:49.925980 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:49.927661 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:49.928083 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:49.928206 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:49.931051 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:49.931058 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:49.931061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:49.931096 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:49.932684 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:49.933419 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:49.933422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:49.933424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:49.933507 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:49.933943 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:49.933946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:49.933947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:49.934130 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:49.934207 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:49.934913 (+ 706us) server_negotiation.cc:299] Negotiation successful
0619 12:35:49.935027 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":51}
I20260619 12:35:49.936486 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:49.922174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.2:47931)
0619 12:35:49.922624 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:49.922667 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:49.923896 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:49.924415 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:49.924422 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:49.924795 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:49.925270 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:49.925282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:49.926120 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:49.926123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:49.927543 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:49.927554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:49.928328 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:49.928332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:49.928524 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:49.929217 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:49.929240 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:49.930858 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:49.932893 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:49.932899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:49.932919 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:49.933306 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:49.933607 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:49.933613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:49.933618 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:49.933835 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:49.934235 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:49.934246 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:49.934642 (+ 396us) client_negotiation.cc:769] Sending connection context
0619 12:35:49.934891 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 12:35:49.935289 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:35:49.937743 6831 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43151
I20260619 12:35:49.937961 6831 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:49.938525 6831 heartbeater.cc:511] Master 127.0.98.62:43151 requested a full tablet report, sending...
I20260619 12:35:49.939579 6506 ts_manager.cc:194] Registered new tserver with Master: f5bb8dd6c140400b91cceefd251251c6 (127.0.98.2:37503)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260619 12:35:49.940318 6506 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 12:35:49.955020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43151
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:50.068262 6836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:50.068584 6836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:50.068668 6836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:50.072813 6836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:50.072911 6836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:50.073001 6836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:35:50.078648 6836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:50.079864 6836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:50.080886 6836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:50.090684 6841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:50.090827 6842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:50.090677 6844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:50.090864 6836 server_base.cc:1034] running on GCE node
I20260619 12:35:50.091581 6836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:50.092249 6836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:50.093463 6836 hybrid_clock.cc:648] HybridClock initialized: now 1781872550093445 us; error 38 us; skew 500 ppm
Jun 19 12:35:50 dist-test-slave-kdrb krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:50.096422 6836 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 12:35:50.097438 6836 webserver.cc:460] Webserver started at http://127.0.98.3:34933/ using document root <none> and password file <none>
I20260619 12:35:50.098027 6836 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:50.098116 6836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:50.098393 6836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:50.100660 6836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "c23456507c944441a0bbe0b9d3cc6e5a"
format_stamp: "Formatted at 2026-06-19 12:35:50 on dist-test-slave-kdrb"
server_key: "cbf28bce6345e1191b867c51cedf10b4"
server_key_iv: "738952dbe424a9cf684c5c3ba7f6617a"
server_key_version: "encryptionkey@0"
I20260619 12:35:50.101265 6836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c23456507c944441a0bbe0b9d3cc6e5a"
format_stamp: "Formatted at 2026-06-19 12:35:50 on dist-test-slave-kdrb"
server_key: "cbf28bce6345e1191b867c51cedf10b4"
server_key_iv: "738952dbe424a9cf684c5c3ba7f6617a"
server_key_version: "encryptionkey@0"
I20260619 12:35:50.105441 6836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:35:50.107976 6851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:50.109270 6836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:35:50.109436 6836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "c23456507c944441a0bbe0b9d3cc6e5a"
format_stamp: "Formatted at 2026-06-19 12:35:50 on dist-test-slave-kdrb"
server_key: "cbf28bce6345e1191b867c51cedf10b4"
server_key_iv: "738952dbe424a9cf684c5c3ba7f6617a"
server_key_version: "encryptionkey@0"
I20260619 12:35:50.109560 6836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:50.126111 6836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:50.129087 6836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:50.129307 6836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:50.129926 6836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:50.130789 6836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:50.130858 6836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:50.130928 6836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:50.130975 6836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:50.139947 6836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33795
I20260619 12:35:50.139966 6964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33795 every 8 connection(s)
I20260619 12:35:50.140875 6836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:35:50.141400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 6836
I20260619 12:35:50.141520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:35:50.141810 392 external_mini_cluster.cc:1428] Setting key e1d8a1e4496fcb3331ac567be4f53a9e
Jun 19 12:35:50 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:35:50.154127 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.142870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50495 (local address 127.0.98.62:43151)
0619 12:35:50.143029 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.143034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.143964 (+ 930us) server_negotiation.cc:407] Connection header received
0619 12:35:50.145337 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.145339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.145393 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.145466 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.146561 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.147109 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.148246 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.148460 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.148546 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.150979 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.150982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.150984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.151009 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.152365 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.152827 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.152829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.152830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.152917 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.153196 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.153198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.153199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.153375 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.153450 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.153905 (+ 455us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.154005 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 12:35:50.155046 6967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.143316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.3:50495)
0619 12:35:50.143822 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:50.143865 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:50.145133 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:50.145600 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:50.145606 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:50.145965 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:50.146435 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.146447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.147214 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.147216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.148120 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.148129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.148586 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.148589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.148728 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.149353 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:50.149369 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:50.150816 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:50.152472 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.152477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.152489 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.152743 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.152991 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.152993 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.152994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.153125 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.153455 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:50.153460 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:50.153694 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 12:35:50.153851 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 12:35:50.154106 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":169,"threads_started":1}
I20260619 12:35:50.156176 6965 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43151
I20260619 12:35:50.156381 6965 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:50.156800 6965 heartbeater.cc:511] Master 127.0.98.62:43151 requested a full tablet report, sending...
I20260619 12:35:50.157721 6505 ts_manager.cc:194] Registered new tserver with Master: c23456507c944441a0bbe0b9d3cc6e5a (127.0.98.3:33795)
I20260619 12:35:50.158265 6505 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50495
I20260619 12:35:50.168156 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260619 12:35:50.194980 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40045
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43151,127.0.98.61:40045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:50.308807 6970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:50.309134 6970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:50.309212 6970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:50.313443 6970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:50.313531 6970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:50.313567 6970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:50.313611 6970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:50.313637 6970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:50.319093 6970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43151,127.0.98.61:40045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
W20260619 12:35:50.319311 6970 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:43151,127.0.98.61:40045'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 12:35:50.320257 6970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:50.321225 6970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:50.329643 6978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:50.329767 6970 server_base.cc:1034] running on GCE node
W20260619 12:35:50.329640 6976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:50.329639 6975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:50.330545 6970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:50.331779 6970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:50.332942 6970 hybrid_clock.cc:648] HybridClock initialized: now 1781872550332924 us; error 33 us; skew 500 ppm
Jun 19 12:35:50 dist-test-slave-kdrb krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:50.335778 6970 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 12:35:50.336634 6970 webserver.cc:460] Webserver started at http://127.0.98.61:35489/ using document root <none> and password file <none>
I20260619 12:35:50.337164 6970 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:50.337222 6970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:50.337385 6970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:50.339461 6970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/data/instance:
uuid: "9704aa957af5431b9e7527b8b3adbe0a"
format_stamp: "Formatted at 2026-06-19 12:35:50 on dist-test-slave-kdrb"
server_key: "0e7557978be82e16c31a391f5334994c"
server_key_iv: "bec8de3eca5ffa67532932d1ad2da30c"
server_key_version: "encryptionkey@0"
I20260619 12:35:50.340057 6970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/wal/instance:
uuid: "9704aa957af5431b9e7527b8b3adbe0a"
format_stamp: "Formatted at 2026-06-19 12:35:50 on dist-test-slave-kdrb"
server_key: "0e7557978be82e16c31a391f5334994c"
server_key_iv: "bec8de3eca5ffa67532932d1ad2da30c"
server_key_version: "encryptionkey@0"
I20260619 12:35:50.343776 6970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 12:35:50.346115 6985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:50.347359 6970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:35:50.347463 6970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/wal
uuid: "9704aa957af5431b9e7527b8b3adbe0a"
format_stamp: "Formatted at 2026-06-19 12:35:50 on dist-test-slave-kdrb"
server_key: "0e7557978be82e16c31a391f5334994c"
server_key_iv: "bec8de3eca5ffa67532932d1ad2da30c"
server_key_version: "encryptionkey@0"
I20260619 12:35:50.347620 6970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:50.358749 6970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:50.362279 6970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:50.362442 6970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:50.369222 6970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40045
I20260619 12:35:50.369285 7037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40045 every 8 connection(s)
I20260619 12:35:50.370208 6970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/data/info.pb
I20260619 12:35:50.371061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 6970
I20260619 12:35:50.373420 7038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:50.378077 7038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:35:50 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 12:35:50 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:35:50.387250 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.372674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45194 (local address 127.0.98.61:40045)
0619 12:35:50.373093 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.373114 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.373158 (+ 44us) server_negotiation.cc:407] Connection header received
0619 12:35:50.373998 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.374021 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.374427 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.374798 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.375546 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.377066 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.378095 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.378769 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.378935 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.382072 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.382077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.382086 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.382111 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.384144 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.384598 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.384602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.384607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.384767 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.385063 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.385066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.385068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.385560 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.385720 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.386067 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.386379 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":116,"threads_started":1}
I20260619 12:35:50.390520 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.380098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.0.98.62:43151)
0619 12:35:50.380238 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.380244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.380602 (+ 358us) server_negotiation.cc:407] Connection header received
0619 12:35:50.380758 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.380761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.380813 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.380908 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.381617 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.382270 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.383850 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.384066 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.384147 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.386642 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.386646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.386648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.386672 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.388728 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.389298 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.389302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.389304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.389411 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.389669 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.389676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.389678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.389888 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.389989 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.390306 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.390408 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 12:35:50.390548 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.380052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.0.1:58810)
0619 12:35:50.380472 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:50.380513 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:50.380661 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:50.380958 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:50.380962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:50.381297 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:50.381492 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.381498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.382659 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.382662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.383737 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.383748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.384173 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.384179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.384303 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.384794 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:50.384812 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:50.386488 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:50.388861 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.388871 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.388876 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.389168 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.389497 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.389499 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.389501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.389577 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.390018 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:50.390023 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:50.390128 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 12:35:50.390249 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 12:35:50.390398 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"spinlock_wait_cycles":2944,"thread_start_us":116,"threads_started":1}
I20260619 12:35:50.391681 7038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:35:50 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 12:35:50.399709 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.392200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45198 (local address 127.0.98.61:40045)
0619 12:35:50.392300 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.392304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.392357 (+ 53us) server_negotiation.cc:407] Connection header received
0619 12:35:50.392495 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.392498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.392534 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.392610 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.392966 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.393462 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.394190 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.394401 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.394503 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.396938 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.396941 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.396943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.396961 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.398137 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.398545 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.398552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.398554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.398623 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.398882 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.398885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.398887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.399057 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.399155 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.399390 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.399583 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":33}
I20260619 12:35:50.399708 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.392160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40045 (local address 127.0.0.1:45198)
0619 12:35:50.392289 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:50.392304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:50.392390 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:50.392619 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:50.392621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:50.392779 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:50.392853 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.392858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.393566 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.393569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.394084 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.394091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.394519 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.394522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.394608 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.395068 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:50.395079 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:50.396799 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:50.398246 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.398252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.398255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.398454 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.398713 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.398716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.398718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.398790 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.399155 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:50.399158 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:50.399241 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 12:35:50.399342 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 12:35:50.399527 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 12:35:50.402837 7038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: Bootstrap starting.
I20260619 12:35:50.405259 7038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:50.405972 7038 log.cc:824] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:50.408010 7038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: No bootstrap required, opened a new log
I20260619 12:35:50.410979 7038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } }
I20260619 12:35:50.411190 7038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:50.411232 7038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9704aa957af5431b9e7527b8b3adbe0a, State: Initialized, Role: FOLLOWER
I20260619 12:35:50.411685 7038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } }
I20260619 12:35:50.412142 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } } }
I20260619 12:35:50.412299 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:35:50.412953 7038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: configured and running, proceeding with master startup.
W20260619 12:35:50.417774 7055 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:35:50.417870 7055 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:35:50.419570 6970 master_runner.cc:428] Detected that this master 9704aa957af5431b9e7527b8b3adbe0a is joining an existing cluster
I20260619 12:35:50.419648 6970 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:40045
I20260619 12:35:50.430511 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.422235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45200 (local address 127.0.98.61:40045)
0619 12:35:50.422333 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.422339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.422749 (+ 410us) server_negotiation.cc:407] Connection header received
0619 12:35:50.422862 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.422865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.422912 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.423007 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.423734 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.424519 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.425562 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.425762 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.425867 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.427069 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.427073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.427079 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.427099 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.428675 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.429078 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.429084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.429086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.429154 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.429429 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.429434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.429436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.429598 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.429952 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.430187 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.430344 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":37}
I20260619 12:35:50.430804 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.422123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.0.98.62:43151)
0619 12:35:50.422257 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.422262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.422741 (+ 479us) server_negotiation.cc:407] Connection header received
0619 12:35:50.422907 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.422910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.422957 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.423128 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.423771 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.424289 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.425454 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.425811 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.425919 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.427298 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.427302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.427305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.427333 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.429036 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.429515 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.429521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.429525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.429608 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.429925 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.429928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.429930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.430128 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.430195 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.430528 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.430642 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 12:35:50.430523 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.422056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40045 (local address 127.0.0.1:45200)
0619 12:35:50.422627 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:50.422642 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:50.422775 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:50.423099 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:50.423106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:50.423402 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:50.423595 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.423614 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.424662 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.424665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.425468 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.425474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.425876 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.425880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.425960 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.426486 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:50.426499 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:50.426916 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:50.428788 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.428794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.428797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.428988 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.429256 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.429260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.429262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.429340 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.429942 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:50.429945 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:50.430040 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 12:35:50.430144 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 12:35:50.430290 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":21,"thread_start_us":113,"threads_started":1}
I20260619 12:35:50.431247 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.422371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.0.1:58814)
0619 12:35:50.422663 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:50.422676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:50.422789 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:50.423265 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:50.423271 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:50.423539 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:50.423675 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.423683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.424396 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.424402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.425324 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.425333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.426038 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.426041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.426159 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.426637 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:50.426652 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:50.427142 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:50.429158 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.429161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.429163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.429401 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.429732 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.429735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.429737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.429819 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.430265 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:50.430268 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:50.430357 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 12:35:50.430945 (+ 588us) client_negotiation.cc:240] Negotiation successful
0619 12:35:50.431074 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":93,"threads_started":1}
Jun 19 12:35:50 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872549, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 12:35:50.450990 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.439928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.0.98.61:40045)
0619 12:35:50.440053 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.440057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.440071 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:35:50.440255 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.440259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.440318 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.440437 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.441449 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.442232 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.443563 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.443911 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.444010 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.446948 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.446952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.446955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.446983 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.448742 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.449319 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.449323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.449325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.449417 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.449691 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.449693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.449695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.449956 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.450034 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.450615 (+ 581us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.450758 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 12:35:50.451790 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.435894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40045 (local address 127.0.0.1:45208)
0619 12:35:50.436261 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:50.436312 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:50.436444 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:50.440609 (+ 4165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:50.440617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:50.441141 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:50.441282 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.441291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.442388 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.442391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.443421 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.443432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.444143 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.444147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.444262 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.444901 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:50.444918 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:50.446754 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:50.448877 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.448890 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.448898 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.449207 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.449519 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.449522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.449524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.449602 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.450114 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:50.450124 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:50.450276 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 12:35:50.451363 (+ 1087us) client_negotiation.cc:240] Negotiation successful
0619 12:35:50.451618 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":99,"threads_started":1}
I20260619 12:35:50.453541 6505 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:40045: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 40045
}
attrs {
promote: true
}
}
dest_uuid: "5a6c5cff1ee44943a5f87b30f25d9083"
cas_config_opid_index: -1
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 12:35:50.454339 6505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: true } }
I20260619 12:35:50.456194 7065 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9704aa957af5431b9e7527b8b3adbe0a (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: true } } }
I20260619 12:35:50.456902 7012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260619 12:35:50.458256 7065 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:40045
I20260619 12:35:50.458729 6970 master.cc:537] Master@127.0.98.61:40045 shutting down...
I20260619 12:35:50.458843 7012 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 1 FOLLOWER]: Refusing update from remote peer 5a6c5cff1ee44943a5f87b30f25d9083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 12:35:50.459085 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: true } } }
I20260619 12:35:50.459200 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:50.459758 7065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 12:35:50.461694 6490 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 12:35:50.462550 6490 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 -> Peer 9704aa957af5431b9e7527b8b3adbe0a (127.0.98.61:40045): Couldn't send request to peer 9704aa957af5431b9e7527b8b3adbe0a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 12:35:50.463796 6970 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:50.464073 6970 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:50.464175 6970 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: stopping tablet replica
I20260619 12:35:50.468981 6970 master.cc:552] Master@127.0.98.61:40045 shutdown complete.
I20260619 12:35:50.469046 6970 master_runner.cc:305] Clearing existing system tablet
I20260619 12:35:50.469260 6970 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:50.469326 6970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:50.471549 6970 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:35:50.472383 7071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:50.472618 6970 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:50.472739 6970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:35:50.472774 6970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/wal
uuid: "9704aa957af5431b9e7527b8b3adbe0a"
format_stamp: "Formatted at 2026-06-19 12:35:50 on dist-test-slave-kdrb"
server_key: "0e7557978be82e16c31a391f5334994c"
server_key_iv: "bec8de3eca5ffa67532932d1ad2da30c"
server_key_version: "encryptionkey@0"
I20260619 12:35:50.474658 6970 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:50.477420 6970 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 12:35:50.477519 6970 log.cc:1196] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260619 12:35:50.477773 6970 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: Deleting consensus metadata
I20260619 12:35:50.477905 6970 master_runner.cc:315] Copying system tablet from 127.0.98.62:43151
I20260619 12:35:50.478714 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42811
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43151,127.0.98.61:40045,127.0.98.60:42811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:35:50.479347 6970 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43151
I20260619 12:35:50.492671 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.480098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58818 (local address 127.0.98.62:43151)
0619 12:35:50.480244 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.480250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.483878 (+ 3628us) server_negotiation.cc:407] Connection header received
0619 12:35:50.484055 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.484059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.484120 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.484227 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.484868 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.485688 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.487083 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.487439 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.487566 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.489085 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.489089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.489092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.489120 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.490860 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.491353 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.491356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.491358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.491448 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.491767 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.491770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.491772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.491998 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.492097 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.492373 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.492504 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 12:35:50.492758 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.479974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.0.1:58818)
0619 12:35:50.483722 (+ 3748us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:50.483742 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:50.483935 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:50.484271 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:50.484276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:50.484531 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:50.484709 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.484720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.485816 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.485823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.486931 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.486945 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.487602 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.487610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.487736 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.488378 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:50.488395 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:50.488913 (+ 518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:50.490976 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.490979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.490982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.491240 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.491582 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.491587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.491589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.491678 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.492092 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:50.492095 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:50.492194 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 12:35:50.492296 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 12:35:50.492453 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3628,"thread_start_us":87,"threads_started":1}
I20260619 12:35:50.493703 6526 tablet_copy_service.cc:140] P 5a6c5cff1ee44943a5f87b30f25d9083: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9704aa957af5431b9e7527b8b3adbe0a ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:58818)
I20260619 12:35:50.493902 6526 tablet_copy_service.cc:161] P 5a6c5cff1ee44943a5f87b30f25d9083: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9704aa957af5431b9e7527b8b3adbe0a at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:58818: session id = 9704aa957af5431b9e7527b8b3adbe0a-00000000000000000000000000000000
I20260619 12:35:50.495747 6526 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 12:35:50.497588 6970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 12:35:50.500057 6970 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: tablet copy: Starting download of 0 data blocks...
I20260619 12:35:50.500268 6970 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: tablet copy: Starting download of 1 WAL segments...
I20260619 12:35:50.502187 6970 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 12:35:50.504177 6526 tablet_copy_service.cc:342] P 5a6c5cff1ee44943a5f87b30f25d9083: Request end of tablet copy session 9704aa957af5431b9e7527b8b3adbe0a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:58818
I20260619 12:35:50.504307 6526 tablet_copy_service.cc:433] P 5a6c5cff1ee44943a5f87b30f25d9083: ending tablet copy session 9704aa957af5431b9e7527b8b3adbe0a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9704aa957af5431b9e7527b8b3adbe0a
I20260619 12:35:50.512526 6970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:50.514312 7082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:50.514325 7079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:50.514601 7080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:50.515053 6970 server_base.cc:1034] running on GCE node
I20260619 12:35:50.515244 6970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:50.515561 6970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:50.516697 6970 hybrid_clock.cc:648] HybridClock initialized: now 1781872550516684 us; error 32 us; skew 500 ppm
Jun 19 12:35:50 dist-test-slave-kdrb krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:50.518328 6970 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 12:35:50.518777 6970 webserver.cc:460] Webserver started at http://127.0.98.61:38365/ using document root <none> and password file <none>
I20260619 12:35:50.518955 6970 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:50.518989 6970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:50.520328 6970 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:35:50.521265 7089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:50.521497 6970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:35:50.521566 6970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/wal
uuid: "9704aa957af5431b9e7527b8b3adbe0a"
format_stamp: "Formatted at 2026-06-19 12:35:50 on dist-test-slave-kdrb"
server_key: "0e7557978be82e16c31a391f5334994c"
server_key_iv: "bec8de3eca5ffa67532932d1ad2da30c"
server_key_version: "encryptionkey@0"
I20260619 12:35:50.521641 6970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:50.558910 6970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:50.561350 6970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:50.566399 6970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40045
I20260619 12:35:50.566524 7141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40045 every 8 connection(s)
I20260619 12:35:50.566982 6970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-1/data/info.pb
I20260619 12:35:50.568814 7142 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:35:50.569940 7142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: Bootstrap starting.
I20260619 12:35:50.581365 7142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:35:50.581979 7142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: Bootstrap complete.
I20260619 12:35:50.582692 7142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: true } }
I20260619 12:35:50.582943 7142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9704aa957af5431b9e7527b8b3adbe0a, State: Initialized, Role: LEARNER
I20260619 12:35:50.583119 7142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: true } }
I20260619 12:35:50.583614 7145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: true } } }
I20260619 12:35:50.583732 7145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: This master's current role is: LEARNER
I20260619 12:35:50.583899 7142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:50.588122 7156 catalog_manager.cc:1259] Loaded cluster ID: 88dd114f19f84679bc91e6d4d7d7c049
I20260619 12:35:50.588191 7156 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: loading cluster ID for follower catalog manager: success
I20260619 12:35:50.590322 7156 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: acquiring CA information for follower catalog manager: success
I20260619 12:35:50.591760 7156 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 12:35:50.616601 7077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:50.616880 7077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:50.616925 7077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:50.620905 7077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:50.620978 7077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:50.620998 7077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:50.621016 7077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:50.621031 7077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:50.626622 7077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43151,127.0.98.61:40045,127.0.98.60:42811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:50.627758 7077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:50.628750 7077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:50.637144 7164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:50.637174 7161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:50.637178 7162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:50.637343 7077 server_base.cc:1034] running on GCE node
I20260619 12:35:50.638126 7077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:50.639258 7077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:50.640440 7077 hybrid_clock.cc:648] HybridClock initialized: now 1781872550640433 us; error 43 us; skew 500 ppm
Jun 19 12:35:50 dist-test-slave-kdrb krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:50.643301 7077 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 12:35:50.644311 7077 webserver.cc:460] Webserver started at http://127.0.98.60:37261/ using document root <none> and password file <none>
I20260619 12:35:50.644876 7077 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:50.644943 7077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:50.645193 7077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:50.647296 7077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/data/instance:
uuid: "ac343c85736a403ab6e799a52157d160"
format_stamp: "Formatted at 2026-06-19 12:35:50 on dist-test-slave-kdrb"
server_key: "447963efb1ae9eb706f5cd6348cda85e"
server_key_iv: "f0d02b68693ef853cdeda588f97ac35a"
server_key_version: "encryptionkey@0"
I20260619 12:35:50.647964 7077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/wal/instance:
uuid: "ac343c85736a403ab6e799a52157d160"
format_stamp: "Formatted at 2026-06-19 12:35:50 on dist-test-slave-kdrb"
server_key: "447963efb1ae9eb706f5cd6348cda85e"
server_key_iv: "f0d02b68693ef853cdeda588f97ac35a"
server_key_version: "encryptionkey@0"
I20260619 12:35:50.651898 7077 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 12:35:50.654277 7171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:50.655597 7077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:35:50.655745 7077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/wal
uuid: "ac343c85736a403ab6e799a52157d160"
format_stamp: "Formatted at 2026-06-19 12:35:50 on dist-test-slave-kdrb"
server_key: "447963efb1ae9eb706f5cd6348cda85e"
server_key_iv: "f0d02b68693ef853cdeda588f97ac35a"
server_key_version: "encryptionkey@0"
I20260619 12:35:50.655879 7077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:50.669303 7077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:50.672732 7077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:50.672932 7077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:50.680399 7077 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42811
I20260619 12:35:50.680689 7223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42811 every 8 connection(s)
I20260619 12:35:50.681393 7077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/data/info.pb
I20260619 12:35:50.684257 7224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:50.685137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 7077
I20260619 12:35:50.690237 7224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:35:50 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 12:35:50 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:35:50.701573 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.686588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56810 (local address 127.0.98.60:42811)
0619 12:35:50.686998 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.687013 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.687049 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:35:50.688039 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.688061 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.688458 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.688772 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.689742 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.690829 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.691944 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.692641 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.693218 (+ 577us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.695686 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.695700 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.695712 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.695751 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.698064 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.698681 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.698688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.698696 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.698830 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.699156 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.699159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.699161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.699683 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.699859 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.700322 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.700641 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":129,"threads_started":1}
I20260619 12:35:50.703116 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.692243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.0.98.62:43151)
0619 12:35:50.692370 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.692373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.692688 (+ 315us) server_negotiation.cc:407] Connection header received
0619 12:35:50.692851 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.692853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.692902 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.692982 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.693770 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.694342 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.695802 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.696074 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.696191 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.699164 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.699168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.699171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.699202 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.701128 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.701695 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.701701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.701704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.701801 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.702112 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.702115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.702117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.702397 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.702506 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.702855 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.702974 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:35:50.703122 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.692106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.0.1:58822)
0619 12:35:50.692576 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:50.692614 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:50.692742 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:50.693028 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:50.693033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:50.693453 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:50.693631 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.693639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.694482 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.694488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.695685 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.695693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.696220 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.696223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.696356 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.697132 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:50.697152 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:50.699001 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:50.701244 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.701251 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.701256 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.701530 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.701916 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.701919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.701921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.702003 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.702507 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:50.702512 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:50.702627 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 12:35:50.702765 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 12:35:50.702941 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":172,"threads_started":1}
I20260619 12:35:50.705061 7224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:35:50 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 12:35:50.713371 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.705569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40045 (local address 127.0.0.1:45216)
0619 12:35:50.705896 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:50.705908 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:50.705970 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:50.706207 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:50.706209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:50.706346 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:50.706438 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.706443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.707326 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.707329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.708056 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.708067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.708544 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.708547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.708623 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.709052 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:50.709064 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:50.710261 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:50.711989 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.711991 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.711993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.712223 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.712500 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.712503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.712504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.712586 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.712942 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:50.712944 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:50.713022 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 12:35:50.713132 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 12:35:50.713242 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266}
I20260619 12:35:50.713397 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.705695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.0.98.61:40045)
0619 12:35:50.705956 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.705960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.705976 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:35:50.706042 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.706045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.706090 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.706179 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.706537 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.707213 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.708173 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.708409 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.708499 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.710399 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.710403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.710406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.710432 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.711873 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.712307 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.712310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.712311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.712386 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.712669 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.712672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.712673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.712860 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.712938 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.713172 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.713274 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
I20260619 12:35:50.714201 7224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:35:50 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 12:35:50.721956 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.714695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42811 (local address 127.0.0.1:56822)
0619 12:35:50.714805 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:50.714820 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:50.714907 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:50.715127 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:50.715130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:50.715292 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:50.715352 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.715357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.716441 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.716447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.716953 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.716959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.717401 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.717404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.717473 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.717906 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:50.717916 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:50.719044 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:50.720535 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.720541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.720544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.720723 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.720978 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.720981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.720983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.721056 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.721426 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:50.721428 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:50.721509 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 12:35:50.721609 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 12:35:50.721746 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 12:35:50.721956 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.714731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.0.98.60:42811)
0619 12:35:50.714817 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.714822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.714869 (+ 47us) server_negotiation.cc:407] Connection header received
0619 12:35:50.715004 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.715007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.715042 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.715120 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.715453 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.716323 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.717068 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.717296 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.717404 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.719170 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.719173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.719176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.719197 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.720431 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.720810 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.720817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.720819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.720889 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.721160 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.721167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.721169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.721322 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.721426 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.721651 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.721823 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":38}
I20260619 12:35:50.725124 7224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: Bootstrap starting.
I20260619 12:35:50.727499 6697 heartbeater.cc:503] Master 127.0.98.62:43151 was elected leader, sending a full tablet report...
I20260619 12:35:50.727569 7224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:50.728325 7224 log.cc:824] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:50.730314 7224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: No bootstrap required, opened a new log
I20260619 12:35:50.733459 7224 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } }
I20260619 12:35:50.733686 7224 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:50.733745 7224 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac343c85736a403ab6e799a52157d160, State: Initialized, Role: FOLLOWER
I20260619 12:35:50.734217 7224 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } }
I20260619 12:35:50.734791 7230 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } } }
I20260619 12:35:50.734961 7230 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:35:50.735648 7224 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 12:35:50.740748 7241 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:35:50.740849 7241 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:35:50.743887 7077 master_runner.cc:428] Detected that this master ac343c85736a403ab6e799a52157d160 is joining an existing cluster
I20260619 12:35:50.743966 7077 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:42811
I20260619 12:35:50.755434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 6568
I20260619 12:35:50.762718 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34107
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43151
--webserver_port=41251
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43151,127.0.98.61:40045
--tserver_master_addrs=127.0.98.62:43151,127.0.98.61:40045,127.0.98.60:42811 with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:35:50.771914 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.746440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.0.98.60:42811)
0619 12:35:50.746567 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.746571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.746780 (+ 209us) server_negotiation.cc:407] Connection header received
0619 12:35:50.746948 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.746951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.746989 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.747055 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.747913 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.748698 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.750136 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.750510 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.750631 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.752177 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.752180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.752183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.752205 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.769888 (+ 17683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.770394 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.770398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.770402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.770480 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.770755 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.770763 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.770765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.771070 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.771148 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.771591 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.771728 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":58}
I20260619 12:35:50.772522 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.746055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42811 (local address 127.0.0.1:56824)
0619 12:35:50.746702 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:50.746717 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:50.746833 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:50.747144 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:50.747147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:50.747549 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:50.747791 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.747799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.748808 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.748811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.750025 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.750034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.750929 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.750933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.751032 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.751605 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:50.751619 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:50.752052 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:50.770013 (+ 17961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.770018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.770023 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.770270 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.770584 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.770589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.770594 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.770664 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.771221 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:50.771226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:50.771330 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 12:35:50.772137 (+ 807us) client_negotiation.cc:240] Negotiation successful
0619 12:35:50.772287 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":33,"spinlock_wait_cycles":1408,"thread_start_us":113,"threads_started":1}
I20260619 12:35:50.774305 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.746451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45224 (local address 127.0.98.61:40045)
0619 12:35:50.746653 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.746658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.746752 (+ 94us) server_negotiation.cc:407] Connection header received
0619 12:35:50.746895 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.746899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.746955 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.747029 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.747933 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.748475 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.749629 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.750038 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.750120 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.768549 (+ 18429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.768559 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.768566 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.768613 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.770643 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.772933 (+ 2290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.772939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.772941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.773040 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.773352 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.773357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.773359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.773582 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.773663 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.774002 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.774147 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260619 12:35:50.774417 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.746061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.0.98.62:43151)
0619 12:35:50.746207 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.746213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.747163 (+ 950us) server_negotiation.cc:407] Connection header received
0619 12:35:50.747326 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.747329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.747376 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.747428 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.748956 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.749728 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.751309 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.751701 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.751819 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.769514 (+ 17695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.769520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.769524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.769561 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.772170 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.773089 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.773093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.773095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.773187 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.773515 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.773518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.773520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.773735 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.773802 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.774144 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.774257 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:35:50.774930 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.746438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.0.1:58824)
0619 12:35:50.747082 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:50.747096 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:50.747222 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:50.747596 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:50.747599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:50.747828 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:50.748823 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.748829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.749869 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.749872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.751184 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.751193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.751827 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.751831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.751934 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.752434 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:50.752453 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:50.768992 (+ 16539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:50.772736 (+ 3744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.772739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.772740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.772969 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.773297 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.773302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.773307 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.773410 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.773866 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:50.773868 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:50.773969 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 12:35:50.774617 (+ 648us) client_negotiation.cc:240] Negotiation successful
0619 12:35:50.774762 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":38,"thread_start_us":82,"threads_started":1}
I20260619 12:35:50.775388 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.746023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40045 (local address 127.0.0.1:45224)
0619 12:35:50.746657 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:50.746673 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:50.746801 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:50.747144 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:50.747147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:50.747545 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:50.747791 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.747799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.748588 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.748590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.749495 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.749503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.750282 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.750285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.750372 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.750925 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:50.750939 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:50.768316 (+ 17377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:50.772388 (+ 4072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.772391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.772393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.772791 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.773158 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.773160 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.773162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.773247 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.773718 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:50.773721 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:50.773810 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 12:35:50.774865 (+ 1055us) client_negotiation.cc:240] Negotiation successful
0619 12:35:50.775220 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":63,"spinlock_wait_cycles":53504,"thread_start_us":117,"threads_started":1}
Jun 19 12:35:50 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872549, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 12:35:50.786849 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.778695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.0.98.60:42811)
0619 12:35:50.778795 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.778799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.778899 (+ 100us) server_negotiation.cc:407] Connection header received
0619 12:35:50.779039 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.779042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.779095 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.779178 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.779702 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.780230 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.781042 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.781266 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.781366 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.783366 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.783370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.783373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.783397 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.785123 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.785560 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.785563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.785565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.785644 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.785934 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.785939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.785941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.786204 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.786302 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.786569 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.786673 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 12:35:50.786906 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.778656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42811 (local address 127.0.0.1:56834)
0619 12:35:50.778821 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:50.778843 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:50.778929 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:50.779192 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:50.779194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:50.779384 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:50.779465 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.779573 (+ 108us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.780345 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.780349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.780926 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.780934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.781381 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.781385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.781493 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.781975 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:50.781989 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:50.783226 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:50.785224 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.785245 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.785247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.785457 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.785730 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.785733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.785735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.785819 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.786304 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:50.786308 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:50.786408 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 12:35:50.786509 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 12:35:50.786671 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 12:35:50.788214 6505 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:42811: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ac343c85736a403ab6e799a52157d160"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 42811
}
attrs {
promote: true
}
}
dest_uuid: "5a6c5cff1ee44943a5f87b30f25d9083"
cas_config_opid_index: 5
I20260619 12:35:50.788811 6505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: true } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } }
I20260619 12:35:50.789896 7065 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ac343c85736a403ab6e799a52157d160 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: true } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } } }
I20260619 12:35:50.791255 7065 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:42811
I20260619 12:35:50.791427 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: true } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } } }
I20260619 12:35:50.791555 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:50.792074 7077 master.cc:537] Master@127.0.98.60:42811 shutting down...
I20260619 12:35:50.792356 7198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:50.793915 7198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 1 FOLLOWER]: Refusing update from remote peer 5a6c5cff1ee44943a5f87b30f25d9083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 12:35:50.795858 7066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:50.797252 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.790025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40045 (local address 127.0.0.1:45226)
0619 12:35:50.790519 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:50.790534 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:50.790612 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:50.793066 (+ 2454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:50.793069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:50.793088 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:50.793176 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.793183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.794207 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.794210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.795555 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.795570 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.796685 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.796689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.796778 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.796955 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 12:35:50.797025 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 12:35:50.797114 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417}
W20260619 12:35:50.797808 6491 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 -> Peer ac343c85736a403ab6e799a52157d160 (127.0.98.60:42811): Couldn't send request to peer ac343c85736a403ab6e799a52157d160. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 12:35:50.798331 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.790292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.0.98.61:40045)
0619 12:35:50.792659 (+ 2367us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.792664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.792680 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:35:50.792737 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.792740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.792810 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.792921 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:50.793302 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.794067 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.795710 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.796340 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.796425 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.796585 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.797453 (+ 868us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.798186 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2256}
I20260619 12:35:50.798758 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 1 LEARNER]: Refusing update from remote peer 5a6c5cff1ee44943a5f87b30f25d9083: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 12:35:50.799484 7066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 12:35:50.801342 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ac343c85736a403ab6e799a52157d160 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: true } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } } }
I20260619 12:35:50.803617 7145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a6c5cff1ee44943a5f87b30f25d9083. Latest consensus state: current_term: 1 leader_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: true } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } } }
I20260619 12:35:50.803735 7145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: This master's current role is: LEARNER
I20260619 12:35:50.803844 7145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: true } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } } }
I20260619 12:35:50.803927 7145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: This master's current role is: LEARNER
I20260619 12:35:50.804764 7065 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083: attempting to promote NON_VOTER 9704aa957af5431b9e7527b8b3adbe0a to VOTER
I20260619 12:35:50.805222 7077 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:50.805289 7065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } }
I20260619 12:35:50.806890 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 1 LEARNER]: Refusing update from remote peer 5a6c5cff1ee44943a5f87b30f25d9083: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260619 12:35:50.807193 6491 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 -> Peer ac343c85736a403ab6e799a52157d160 (127.0.98.60:42811): Couldn't send request to peer ac343c85736a403ab6e799a52157d160. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 12:35:50.806063 7077 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:50.807297 7066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 12:35:50.807528 7077 tablet_replica.cc:330] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: stopping tablet replica
I20260619 12:35:50.810612 7066 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9704aa957af5431b9e7527b8b3adbe0a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } } }
I20260619 12:35:50.812031 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9704aa957af5431b9e7527b8b3adbe0a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } } }
I20260619 12:35:50.812292 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } } }
I20260619 12:35:50.812408 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:50.812659 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } } }
I20260619 12:35:50.812740 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:50.813903 7252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } } }
I20260619 12:35:50.814049 7252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:35:50.815819 7077 master.cc:552] Master@127.0.98.60:42811 shutdown complete.
I20260619 12:35:50.815887 7077 master_runner.cc:305] Clearing existing system tablet
I20260619 12:35:50.816118 7077 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:50.816183 7077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:50.817231 7077 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:35:50.818025 7255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:50.818241 7077 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:50.818399 7077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:35:50.818436 7077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/wal
uuid: "ac343c85736a403ab6e799a52157d160"
format_stamp: "Formatted at 2026-06-19 12:35:50 on dist-test-slave-kdrb"
server_key: "447963efb1ae9eb706f5cd6348cda85e"
server_key_iv: "f0d02b68693ef853cdeda588f97ac35a"
server_key_version: "encryptionkey@0"
I20260619 12:35:50.820101 7077 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:50.822188 7077 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 12:35:50.822294 7077 log.cc:1196] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 12:35:50.822532 7077 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: Deleting consensus metadata
I20260619 12:35:50.822721 7077 master_runner.cc:315] Copying system tablet from 127.0.98.62:43151
I20260619 12:35:50.823859 7077 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43151
I20260619 12:35:50.835024 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.824519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58828 (local address 127.0.98.62:43151)
0619 12:35:50.824648 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:50.824652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:50.827684 (+ 3032us) server_negotiation.cc:407] Connection header received
0619 12:35:50.827839 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:50.827842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:50.827900 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:50.827993 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:50.828462 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.829002 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.829929 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.830170 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.830283 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.831520 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:50.831523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:50.831526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:50.831554 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:50.833131 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.833640 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.833644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.833645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.833735 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:50.834005 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:50.834010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:50.834012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:50.834209 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:50.834315 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:50.834725 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 12:35:50.834856 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:35:50.835033 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:50.824401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.0.1:58828)
0619 12:35:50.827568 (+ 3167us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:50.827589 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:50.827735 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:50.828012 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:50.828015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:50.828237 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:50.828331 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.828338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.829118 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.829123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.829801 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:50.829810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:50.830288 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:50.830291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:50.830387 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:50.830975 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:50.830994 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:50.831342 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:50.833258 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.833262 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.833264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.833529 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.833826 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:50.833830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:50.833832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:50.833916 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:50.834301 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:50.834304 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:50.834418 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 12:35:50.834642 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 12:35:50.834824 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3043,"thread_start_us":87,"threads_started":1}
I20260619 12:35:50.835886 6526 tablet_copy_service.cc:140] P 5a6c5cff1ee44943a5f87b30f25d9083: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ac343c85736a403ab6e799a52157d160 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:58828)
I20260619 12:35:50.836016 6526 tablet_copy_service.cc:161] P 5a6c5cff1ee44943a5f87b30f25d9083: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ac343c85736a403ab6e799a52157d160 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:58828: session id = ac343c85736a403ab6e799a52157d160-00000000000000000000000000000000
I20260619 12:35:50.837198 6526 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 12:35:50.838495 7077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 12:35:50.840577 7077 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: tablet copy: Starting download of 0 data blocks...
I20260619 12:35:50.840804 7077 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: tablet copy: Starting download of 1 WAL segments...
I20260619 12:35:50.842296 7077 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 12:35:50.844067 6526 tablet_copy_service.cc:342] P 5a6c5cff1ee44943a5f87b30f25d9083: Request end of tablet copy session ac343c85736a403ab6e799a52157d160-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:58828
I20260619 12:35:50.844167 6526 tablet_copy_service.cc:433] P 5a6c5cff1ee44943a5f87b30f25d9083: ending tablet copy session ac343c85736a403ab6e799a52157d160-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ac343c85736a403ab6e799a52157d160
I20260619 12:35:50.851521 7077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:50.853156 7262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:50.853156 7265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:50.853704 7077 server_base.cc:1034] running on GCE node
W20260619 12:35:50.857945 7263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:50.858224 7077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:50.858585 7077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:50.859740 7077 hybrid_clock.cc:648] HybridClock initialized: now 1781872550859731 us; error 29 us; skew 500 ppm
Jun 19 12:35:50 dist-test-slave-kdrb krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:50.861445 7077 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 12:35:50.861967 7077 webserver.cc:460] Webserver started at http://127.0.98.60:42993/ using document root <none> and password file <none>
I20260619 12:35:50.862191 7077 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:50.862253 7077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:50.863387 7077 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:35:50.864358 7272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:50.864584 7077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:35:50.864675 7077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/wal
uuid: "ac343c85736a403ab6e799a52157d160"
format_stamp: "Formatted at 2026-06-19 12:35:50 on dist-test-slave-kdrb"
server_key: "447963efb1ae9eb706f5cd6348cda85e"
server_key_iv: "f0d02b68693ef853cdeda588f97ac35a"
server_key_version: "encryptionkey@0"
I20260619 12:35:50.864779 7077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:50.913486 7077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:50.915642 7077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:50.921023 7077 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42811
I20260619 12:35:50.921847 7077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/master-2/data/info.pb
W20260619 12:35:50.925688 7250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:50.925994 7250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:50.926107 7250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 12:35:50.926441 7327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42811 every 8 connection(s)
I20260619 12:35:50.928057 7328 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:35:50.929147 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: Bootstrap starting.
W20260619 12:35:50.932396 7250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:50.932487 7250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:50.932581 7250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:35:50.939064 7250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41251
--tserver_master_addrs=127.0.98.62:43151,127.0.98.61:40045,127.0.98.60:42811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:50.940189 7250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:50.941309 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:35:50.941337 7250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:35:50.941617 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: Bootstrap complete.
I20260619 12:35:50.942193 7328 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } }
I20260619 12:35:50.942332 7328 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ac343c85736a403ab6e799a52157d160, State: Initialized, Role: LEARNER
I20260619 12:35:50.942341 6831 heartbeater.cc:503] Master 127.0.98.62:43151 was elected leader, sending a full tablet report...
I20260619 12:35:50.942425 7328 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } }
I20260619 12:35:50.942888 7328 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:50.943993 7331 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } } }
I20260619 12:35:50.944363 7331 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [sys.catalog]: This master's current role is: LEARNER
I20260619 12:35:50.945604 7342 catalog_manager.cc:1259] Loaded cluster ID: 88dd114f19f84679bc91e6d4d7d7c049
I20260619 12:35:50.945691 7342 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: loading cluster ID for follower catalog manager: success
I20260619 12:35:50.947782 7342 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: acquiring CA information for follower catalog manager: success
I20260619 12:35:50.949235 7342 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 12:35:50.953016 7345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:50.953068 7344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:50.953042 7347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:50.953109 7250 server_base.cc:1034] running on GCE node
I20260619 12:35:50.953649 7250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:50.954326 7250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:50.955574 7250 hybrid_clock.cc:648] HybridClock initialized: now 1781872550955544 us; error 30 us; skew 500 ppm
Jun 19 12:35:50 dist-test-slave-kdrb krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:50.958386 7250 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 12:35:50.959348 7250 webserver.cc:460] Webserver started at http://127.0.98.1:41251/ using document root <none> and password file <none>
I20260619 12:35:50.959980 7250 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:50.960078 7250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:50.964361 7250 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:35:50.967072 7354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:50.968403 7250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 12:35:50.968539 7250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "114c34026d3c4740a1794ba290c620f5"
format_stamp: "Formatted at 2026-06-19 12:35:49 on dist-test-slave-kdrb"
server_key: "a785c40fb270b989187e64a1d865711e"
server_key_iv: "7698f05eababe07077f93085a9f85bb4"
server_key_version: "encryptionkey@0"
I20260619 12:35:50.969112 7250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:50.997265 7250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:51.000489 7250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:51.000695 7250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:51.001309 7250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:51.002271 7250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:51.002319 7250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:51.002385 7250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:51.002422 7250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:51.011435 7250 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34107
I20260619 12:35:51.011503 7467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34107 every 8 connection(s)
I20260619 12:35:51.012538 7250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:35:51.020412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 7250
I20260619 12:35:51.020634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 6702
Jun 19 12:35:51 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 12:35:51 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 12:35:51 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872550, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:35:51.029624 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.014443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43547 (local address 127.0.98.60:42811)
0619 12:35:51.014670 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.014675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.016097 (+ 1422us) server_negotiation.cc:407] Connection header received
0619 12:35:51.017055 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.017059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.017114 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.017174 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:51.018506 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.019537 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.020461 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.020822 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.020917 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.024981 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:51.024985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:51.024988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:51.025016 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:51.026946 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.027691 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.027695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.027697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.027786 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.028190 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.028193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.028195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.028413 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:51.028495 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.029317 (+ 822us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.029450 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
I20260619 12:35:51.029621 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.015094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48097 (local address 127.0.98.61:40045)
0619 12:35:51.015222 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.015226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.016098 (+ 872us) server_negotiation.cc:407] Connection header received
0619 12:35:51.017019 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.017022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.017083 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.017182 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:51.018468 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.019272 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.020648 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.020984 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.021075 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.024981 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:51.024985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:51.024988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:51.025016 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:51.027256 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.027800 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.027804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.027808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.027905 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.028190 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.028193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.028195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.028412 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:51.028495 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.029317 (+ 822us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.029448 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 12:35:51.029798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37503
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43151
--webserver_port=42101
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43151,127.0.98.61:40045
--tserver_master_addrs=127.0.98.62:43151,127.0.98.61:40045,127.0.98.60:42811 with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:35:51.030306 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.014372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57647 (local address 127.0.98.62:43151)
0619 12:35:51.014518 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.014526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.016097 (+ 1571us) server_negotiation.cc:407] Connection header received
0619 12:35:51.017058 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.017061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.017114 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.017171 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:51.018502 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.019026 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.022286 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.022710 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.022803 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.026358 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:51.026364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:51.026371 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:51.026407 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:51.028325 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.028812 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.028819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.028823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.028925 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.029234 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.029240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.029245 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.029462 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:51.029544 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.029933 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.030067 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:35:51.031584 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.014881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42811 (local address 127.0.98.1:43547)
0619 12:35:51.015964 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:51.016006 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:51.016857 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:51.017350 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:51.017360 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:51.017839 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:51.018343 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.018351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.019633 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.019638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.020341 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.020351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.021087 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.021092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.021295 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.022118 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:51.022146 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:51.024775 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:51.027092 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.027103 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.027124 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.027570 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.027899 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.027902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.027904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.028085 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.028567 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:51.028574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:51.029007 (+ 433us) client_negotiation.cc:769] Sending connection context
0619 12:35:51.029842 (+ 835us) client_negotiation.cc:240] Negotiation successful
0619 12:35:51.030354 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":265,"spinlock_wait_cycles":69376,"thread_start_us":135,"threads_started":1}
I20260619 12:35:51.031584 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.014899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40045 (local address 127.0.98.1:48097)
0619 12:35:51.015962 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:51.016002 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:51.016823 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:51.017360 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:51.017363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:51.017766 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:51.018328 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.018348 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.019389 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.019393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.020474 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.020547 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.021183 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.021187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.021368 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.022180 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:51.022191 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:51.024775 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:51.027388 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.027392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.027396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.027693 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.028006 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.028009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.028011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.028087 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.028567 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:51.028574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:51.029007 (+ 433us) client_negotiation.cc:769] Sending connection context
0619 12:35:51.029854 (+ 847us) client_negotiation.cc:240] Negotiation successful
0619 12:35:51.030395 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"mutex_wait_us":439,"spinlock_wait_cycles":143232,"thread_start_us":146,"threads_started":1}
I20260619 12:35:51.031585 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.014881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.1:57647)
0619 12:35:51.015964 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:51.016006 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:51.016857 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:51.017350 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:51.017360 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:51.017839 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:51.018342 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.018351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.019163 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.019166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.022137 (+ 2971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.022151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.023001 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.023007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.023134 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.024427 (+ 1293us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:51.024447 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:51.026186 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:51.028464 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.028470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.028475 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.028701 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.029036 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.029042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.029047 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.029137 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.029605 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:51.029611 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:51.029720 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 12:35:51.030445 (+ 725us) client_negotiation.cc:240] Negotiation successful
0619 12:35:51.030622 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":322,"spinlock_wait_cycles":22272,"thread_start_us":104,"threads_started":1}
I20260619 12:35:51.033015 7470 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40045
I20260619 12:35:51.033030 7468 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42811
I20260619 12:35:51.033283 7468 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:51.033285 7470 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:51.033382 7469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43151
I20260619 12:35:51.033452 7469 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:51.033980 7469 heartbeater.cc:511] Master 127.0.98.62:43151 requested a full tablet report, sending...
I20260619 12:35:51.033968 7470 heartbeater.cc:511] Master 127.0.98.61:40045 requested a full tablet report, sending...
I20260619 12:35:51.033968 7468 heartbeater.cc:511] Master 127.0.98.60:42811 requested a full tablet report, sending...
I20260619 12:35:51.035465 6505 ts_manager.cc:194] Re-registered known tserver with Master: 114c34026d3c4740a1794ba290c620f5 (127.0.98.1:34107)
I20260619 12:35:51.035751 7106 ts_manager.cc:194] Registered new tserver with Master: 114c34026d3c4740a1794ba290c620f5 (127.0.98.1:34107)
I20260619 12:35:51.035960 7289 ts_manager.cc:194] Registered new tserver with Master: 114c34026d3c4740a1794ba290c620f5 (127.0.98.1:34107)
I20260619 12:35:51.036453 6505 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57647
W20260619 12:35:51.149307 7476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:51.149626 7476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:51.149700 7476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:51.153887 7476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:51.153956 7476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:51.154055 7476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:35:51.159435 7476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42101
--tserver_master_addrs=127.0.98.62:43151,127.0.98.61:40045,127.0.98.60:42811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:51.160235 6965 heartbeater.cc:503] Master 127.0.98.62:43151 was elected leader, sending a full tablet report...
I20260619 12:35:51.160851 7476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:51.162091 7476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:51.172027 7482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:51.172035 7484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:51.172045 7481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:51.172356 7476 server_base.cc:1034] running on GCE node
I20260619 12:35:51.172850 7476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:51.173538 7476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:51.174734 7476 hybrid_clock.cc:648] HybridClock initialized: now 1781872551174705 us; error 44 us; skew 500 ppm
Jun 19 12:35:51 dist-test-slave-kdrb krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872551, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:51.177520 7476 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 12:35:51.178453 7476 webserver.cc:460] Webserver started at http://127.0.98.2:42101/ using document root <none> and password file <none>
I20260619 12:35:51.179042 7476 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:51.179153 7476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:51.183550 7476 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:35:51.185976 7491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:51.187678 7476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:35:51.187813 7476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "f5bb8dd6c140400b91cceefd251251c6"
format_stamp: "Formatted at 2026-06-19 12:35:49 on dist-test-slave-kdrb"
server_key: "0387c19c7419ca6646bfb6320cea5a32"
server_key_iv: "6b37b4bee382cbd0985db7639e98d95d"
server_key_version: "encryptionkey@0"
I20260619 12:35:51.188453 7476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:51.190665 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.186826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.0.98.60:42811)
0619 12:35:51.187156 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.187159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.187171 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:35:51.187214 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.187216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.187263 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.187386 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:51.187682 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.188243 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.189470 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.190057 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.190163 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.190335 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.190495 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.190564 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260619 12:35:51.190699 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.186765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42811 (local address 127.0.0.1:56850)
0619 12:35:51.186915 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:51.186930 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:51.187023 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:51.187413 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:51.187415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:51.187434 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:51.187572 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.187577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.188364 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.188371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.189313 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.189322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.190244 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.190248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.190349 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.190377 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 12:35:51.190478 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 12:35:51.190562 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 12:35:51.191308 7299 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 12:35:51.191793 7331 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a6c5cff1ee44943a5f87b30f25d9083. Latest consensus state: current_term: 1 leader_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } } }
I20260619 12:35:51.191955 7331 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [sys.catalog]: This master's current role is: LEARNER
I20260619 12:35:51.201342 7476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:51.204710 7476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:51.204891 7476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:51.205475 7476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:51.206337 7476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:51.206393 7476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:51.206434 7476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:51.206450 7476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:51.215634 7476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37503
I20260619 12:35:51.215698 7604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37503 every 8 connection(s)
I20260619 12:35:51.216691 7476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:35:51.220852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 7476
I20260619 12:35:51.221086 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 6836
Jun 19 12:35:51 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872551, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 12:35:51 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872551, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 12:35:51.228751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33795
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43151
--webserver_port=34933
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43151,127.0.98.61:40045
--tserver_master_addrs=127.0.98.62:43151,127.0.98.61:40045,127.0.98.60:42811 with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:35:51.233593 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.218515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39953 (local address 127.0.98.60:42811)
0619 12:35:51.218675 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.218680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.220217 (+ 1537us) server_negotiation.cc:407] Connection header received
0619 12:35:51.221208 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.221214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.221270 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.221350 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:51.222960 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.223631 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.224868 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.225099 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.225208 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.228780 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:51.228784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:51.228798 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:51.228824 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:51.231455 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.231991 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.231996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.231999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.232087 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.232441 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.232445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.232448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.232684 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:51.232760 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.233268 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.233403 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 12:35:51 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872551, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 12:35:51.233668 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.218677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50815 (local address 127.0.98.62:43151)
0619 12:35:51.218805 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.218809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.219939 (+ 1130us) server_negotiation.cc:407] Connection header received
0619 12:35:51.220897 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.220900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.220961 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.221039 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:51.222970 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.223940 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.225413 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.225759 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.225880 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.228780 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:51.228784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:51.228787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:51.228817 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:51.230901 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.231628 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.231633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.231635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.231736 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.232188 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.232191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.232192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.232450 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:51.232545 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.233273 (+ 728us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.233427 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 12:35:51.235597 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.219114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42811 (local address 127.0.98.2:39953)
0619 12:35:51.220105 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:51.220117 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:51.220698 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:51.221482 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:51.221487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:51.221893 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:51.222789 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.222812 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.223772 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.223775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.224739 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.224748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.225244 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.225248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.225459 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.226178 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:51.226205 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:51.228578 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:51.231600 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.231604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.231606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.231882 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.232212 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.232216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.232219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.232307 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.232848 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:51.232851 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:51.232971 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 12:35:51.233868 (+ 897us) client_negotiation.cc:240] Negotiation successful
0619 12:35:51.234263 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":139,"spinlock_wait_cycles":23808,"thread_start_us":135,"threads_started":1}
I20260619 12:35:51.235597 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.219070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.2:50815)
0619 12:35:51.219770 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:51.219836 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:51.220685 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:51.221231 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:51.221243 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:51.221893 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:51.222790 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.222812 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.224087 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.224093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.225240 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.225253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.225879 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.225882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.225979 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.226477 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:51.226496 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:51.228578 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:51.231027 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.231037 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.231058 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.231497 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.231856 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.231861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.231864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.232059 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.232616 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:51.232625 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:51.232969 (+ 344us) client_negotiation.cc:769] Sending connection context
0619 12:35:51.234435 (+ 1466us) client_negotiation.cc:240] Negotiation successful
0619 12:35:51.234615 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":84,"spinlock_wait_cycles":14976,"thread_start_us":122,"threads_started":1}
I20260619 12:35:51.237056 7606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43151
I20260619 12:35:51.237334 7606 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:51.237986 7606 heartbeater.cc:511] Master 127.0.98.62:43151 requested a full tablet report, sending...
I20260619 12:35:51.238210 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.218968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56115 (local address 127.0.98.61:40045)
0619 12:35:51.219111 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.219115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.220477 (+ 1362us) server_negotiation.cc:407] Connection header received
0619 12:35:51.228462 (+ 7985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.228466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.228534 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.228591 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:51.229284 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.230075 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.231269 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.231658 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.231739 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.234152 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:51.234156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:51.234159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:51.234187 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:51.235913 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.236999 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.237002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.237004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.237096 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.237411 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.237414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.237416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.237596 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:51.237701 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.237960 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.238076 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:35:51.238210 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.219337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40045 (local address 127.0.98.2:56115)
0619 12:35:51.220383 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:51.220394 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:51.228302 (+ 7908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:51.228750 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:51.228752 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:51.228980 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:51.229145 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.229155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.230191 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.230197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.231120 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.231131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.231919 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.231926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.232024 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.232550 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:51.232564 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:51.234001 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:51.236049 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.236052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.236054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.236881 (+ 827us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.237212 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.237217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.237221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.237309 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.237691 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:51.237694 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:51.237798 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 12:35:51.237900 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 12:35:51.238039 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"thread_start_us":93,"threads_started":1}
I20260619 12:35:51.237053 7605 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42811
I20260619 12:35:51.238680 7605 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:51.238749 7607 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40045
I20260619 12:35:51.238826 7607 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:51.238875 7605 heartbeater.cc:511] Master 127.0.98.60:42811 requested a full tablet report, sending...
I20260619 12:35:51.238955 7607 heartbeater.cc:511] Master 127.0.98.61:40045 requested a full tablet report, sending...
I20260619 12:35:51.239759 7106 ts_manager.cc:194] Registered new tserver with Master: f5bb8dd6c140400b91cceefd251251c6 (127.0.98.2:37503)
I20260619 12:35:51.239759 7289 ts_manager.cc:194] Registered new tserver with Master: f5bb8dd6c140400b91cceefd251251c6 (127.0.98.2:37503)
I20260619 12:35:51.240142 6505 ts_manager.cc:194] Re-registered known tserver with Master: f5bb8dd6c140400b91cceefd251251c6 (127.0.98.2:37503)
I20260619 12:35:51.241016 6505 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50815
W20260619 12:35:51.349732 7612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:51.350056 7612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:51.350148 7612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:51.354348 7612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:51.354453 7612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:51.354542 7612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:35:51.360042 7612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34933
--tserver_master_addrs=127.0.98.62:43151,127.0.98.61:40045,127.0.98.60:42811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:51.361265 7612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:51.362349 7612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:51.372264 7620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:51.372263 7617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:51.372581 7612 server_base.cc:1034] running on GCE node
W20260619 12:35:51.372311 7618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:51.373149 7612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:51.373824 7612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:51.374979 7612 hybrid_clock.cc:648] HybridClock initialized: now 1781872551374961 us; error 31 us; skew 500 ppm
Jun 19 12:35:51 dist-test-slave-kdrb krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872551, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:51.377866 7612 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 12:35:51.378809 7612 webserver.cc:460] Webserver started at http://127.0.98.3:34933/ using document root <none> and password file <none>
I20260619 12:35:51.379387 7612 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:51.379510 7612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:51.383541 7612 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 12:35:51.385721 7627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:51.387012 7612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:35:51.387138 7612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "c23456507c944441a0bbe0b9d3cc6e5a"
format_stamp: "Formatted at 2026-06-19 12:35:50 on dist-test-slave-kdrb"
server_key: "cbf28bce6345e1191b867c51cedf10b4"
server_key_iv: "738952dbe424a9cf684c5c3ba7f6617a"
server_key_version: "encryptionkey@0"
I20260619 12:35:51.387732 7612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:51.399004 7612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:51.402161 7612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:51.402359 7612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:51.403003 7612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:35:51.403918 7612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:51.403963 7612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:51.404029 7612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:51.404060 7612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:51.413237 7612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33795
I20260619 12:35:51.413259 7740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33795 every 8 connection(s)
I20260619 12:35:51.414345 7612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:35:51.417074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 7612
Jun 19 12:35:51 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872551, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 12:35:51 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872551, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 12:35:51.429772 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.419689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45230 (local address 127.0.98.61:40045)
0619 12:35:51.420203 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.420207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.420220 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:35:51.420254 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.420257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.420304 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.420409 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:51.421721 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.422565 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.423804 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.424157 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.424229 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.426081 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:51.426085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:51.426088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:51.426112 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:51.427851 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.428292 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.428297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.428299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.428407 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.428708 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.428712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.428714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.428952 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:51.429062 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.429442 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.429566 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":70,"threads_started":1}
Jun 19 12:35:51 dist-test-slave-kdrb krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872551, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:35:51.434679 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.416287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42883 (local address 127.0.98.60:42811)
0619 12:35:51.416538 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.416542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.418671 (+ 2129us) server_negotiation.cc:407] Connection header received
0619 12:35:51.419316 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.419320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.419375 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.419434 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:51.424356 (+ 4922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.425075 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.426254 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.426612 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.426684 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.429712 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:51.429716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:51.429718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:51.429743 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:51.431766 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.432580 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.432583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.432590 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.432660 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.433129 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.433132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.433134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.433355 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:51.433435 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.434218 (+ 783us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.434338 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"spinlock_wait_cycles":1792}
I20260619 12:35:51.436020 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.419368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.0.98.60:42811)
0619 12:35:51.420273 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.420284 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.420733 (+ 449us) server_negotiation.cc:407] Connection header received
0619 12:35:51.420775 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.420779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.420826 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.420923 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:51.422304 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.423119 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.428337 (+ 5218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.428708 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.428809 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.430704 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:51.430707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:51.430709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:51.430736 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:51.433591 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.434103 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.434107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.434108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.434195 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.434477 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.434480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.434482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.434680 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:51.435016 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.435633 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.435881 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"spinlock_wait_cycles":3968,"thread_start_us":107,"threads_started":1}
I20260619 12:35:51.436618 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.416698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42811 (local address 127.0.98.3:42883)
0619 12:35:51.417919 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:51.417975 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:51.419058 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:51.419894 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:51.419904 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:51.423277 (+ 3373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:51.423402 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.423409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.425299 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.425302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.426127 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.426136 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.426838 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.426843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.426971 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.427573 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:51.427591 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:51.429549 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:51.431935 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.431944 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.431964 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.432375 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.432777 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.432780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.432782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.433003 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.433527 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:51.433534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:51.433900 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 12:35:51.434949 (+ 1049us) client_negotiation.cc:240] Negotiation successful
0619 12:35:51.435318 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":184,"spinlock_wait_cycles":38016,"thread_start_us":106,"threads_started":1}
I20260619 12:35:51.438445 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.416240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55445 (local address 127.0.98.62:43151)
0619 12:35:51.416445 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.416450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.418078 (+ 1628us) server_negotiation.cc:407] Connection header received
0619 12:35:51.419862 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.419866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.419918 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.420000 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:51.422024 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.422808 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.425887 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.426205 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.426290 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.431534 (+ 5244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:51.431538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:51.431540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:51.431569 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:51.433536 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.433987 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.433991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.433992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.434095 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.437346 (+ 3251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.437351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.437355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.437531 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:51.437647 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.438183 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.438291 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 12:35:51.438635 7741 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42811
I20260619 12:35:51.438879 7741 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:51.439054 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.419112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.0.98.62:43151)
0619 12:35:51.420629 (+ 1517us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.420633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.420879 (+ 246us) server_negotiation.cc:407] Connection header received
0619 12:35:51.421042 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.421045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.421093 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.421160 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:51.423613 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.424369 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.431404 (+ 7035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.431824 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.431947 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.434896 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:51.434899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:51.434902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:51.434925 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:51.436593 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.437177 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.437180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.437182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.437268 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.437893 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.437896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.437898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.438125 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:51.438242 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.438671 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.438822 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1436,"thread_start_us":95,"threads_started":1}
I20260619 12:35:51.439424 7741 heartbeater.cc:511] Master 127.0.98.60:42811 requested a full tablet report, sending...
I20260619 12:35:51.440766 7289 ts_manager.cc:194] Registered new tserver with Master: c23456507c944441a0bbe0b9d3cc6e5a (127.0.98.3:33795)
I20260619 12:35:51.440943 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.416698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.3:55445)
0619 12:35:51.417919 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:51.417975 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:51.419712 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:51.420051 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:51.420054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:51.421642 (+ 1588us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:51.421839 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.421861 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.422941 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.422944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.425754 (+ 2810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.425764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.426462 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.426465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.426595 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.427369 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:51.427396 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:51.431271 (+ 3875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:51.433642 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.433644 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.433646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.433866 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.434204 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.434207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.434210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.437229 (+ 3019us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.437909 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:51.437911 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:51.437999 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 12:35:51.440165 (+ 2166us) client_negotiation.cc:240] Negotiation successful
0619 12:35:51.440310 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"mutex_wait_us":1683,"spinlock_wait_cycles":1152,"thread_start_us":172,"threads_started":1}
I20260619 12:35:51.441524 7742 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43151
I20260619 12:35:51.441819 7742 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:51.442049 7742 heartbeater.cc:511] Master 127.0.98.62:43151 requested a full tablet report, sending...
I20260619 12:35:51.442667 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.416735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38877 (local address 127.0.98.61:40045)
0619 12:35:51.416889 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.416894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.418621 (+ 1727us) server_negotiation.cc:407] Connection header received
0619 12:35:51.419843 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.419847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.419899 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.419943 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:51.422006 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.422695 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.424551 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.424844 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.424938 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.429197 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:51.429201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:51.429203 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:51.429226 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:51.433129 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.437626 (+ 4497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.437629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.437631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.437723 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.439010 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.439014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.439016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.439418 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:51.439721 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.442326 (+ 2605us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.442524 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 12:35:51.442750 6505 ts_manager.cc:194] Re-registered known tserver with Master: c23456507c944441a0bbe0b9d3cc6e5a (127.0.98.3:33795)
I20260619 12:35:51.443166 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.416984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40045 (local address 127.0.98.3:38877)
0619 12:35:51.418507 (+ 1523us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:51.418521 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:51.419613 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:51.420088 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:51.420093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:51.420570 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:51.421876 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.421887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.422890 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.422893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.424427 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.424436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.425134 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.425138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.425400 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.426882 (+ 1482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:51.426908 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:51.429024 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:51.433983 (+ 4959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.433986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.433988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.437516 (+ 3528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.438766 (+ 1250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:51.438772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:51.438777 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:51.438867 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:51.439915 (+ 1048us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:51.439918 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:51.441697 (+ 1779us) client_negotiation.cc:769] Sending connection context
0619 12:35:51.442848 (+ 1151us) client_negotiation.cc:240] Negotiation successful
0619 12:35:51.443001 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1468,"mutex_wait_us":12,"thread_start_us":80,"threads_started":1}
I20260619 12:35:51.443426 6505 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55445
I20260619 12:35:51.443982 7743 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40045
I20260619 12:35:51.444052 7743 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:51.444199 7743 heartbeater.cc:511] Master 127.0.98.61:40045 requested a full tablet report, sending...
I20260619 12:35:51.444819 7106 ts_manager.cc:194] Registered new tserver with Master: c23456507c944441a0bbe0b9d3cc6e5a (127.0.98.3:33795)
I20260619 12:35:51.442960 6506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:35:51.446936 6506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:51.449395 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: true } } }
I20260619 12:35:51.449568 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:51.453316 7065 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083: attempting to promote NON_VOTER ac343c85736a403ab6e799a52157d160 to VOTER
I20260619 12:35:51.453693 7065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } }
I20260619 12:35:51.454840 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 1 FOLLOWER]: Refusing update from remote peer 5a6c5cff1ee44943a5f87b30f25d9083: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 12:35:51.455507 7298 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 1 LEARNER]: Refusing update from remote peer 5a6c5cff1ee44943a5f87b30f25d9083: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 12:35:51.455866 7760 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 12:35:51.456069 7759 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 12:35:51.457736 7759 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ac343c85736a403ab6e799a52157d160 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } } }
I20260619 12:35:51.458124 7298 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ac343c85736a403ab6e799a52157d160 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } } }
I20260619 12:35:51.458607 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ac343c85736a403ab6e799a52157d160 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } } }
I20260619 12:35:51.459122 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } } }
I20260619 12:35:51.459223 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:51.459244 7760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } } }
I20260619 12:35:51.459326 7760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:51.459329 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } } }
I20260619 12:35:51.459427 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:51.460268 7331 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } } }
I20260619 12:35:51.460382 7331 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:35:51.461124 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } } }
I20260619 12:35:51.461261 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:35:51.467815 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.463024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52408 (local address 127.0.98.3:33795)
0619 12:35:51.463316 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.463329 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.463354 (+ 25us) server_negotiation.cc:407] Connection header received
0619 12:35:51.463437 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.463441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.463747 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.463937 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:51.464357 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.465372 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.466587 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.467191 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.467319 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.467468 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.467593 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.467688 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":111,"threads_started":1}
I20260619 12:35:51.467800 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.462874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33795 (local address 127.0.0.1:52408)
0619 12:35:51.462999 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:51.463014 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:51.463096 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:51.464111 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:51.464113 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:51.464131 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:51.464250 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.464257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.465470 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.465473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.466483 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.466489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.467316 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.467320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.467416 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.467443 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 12:35:51.467563 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 12:35:51.467665 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":20}
I20260619 12:35:51.469512 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.463420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34107 (local address 127.0.0.1:48380)
0619 12:35:51.463863 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:51.463877 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:51.463953 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:51.464647 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:51.464650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:51.464669 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:51.464757 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.464763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.466648 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.466651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.467819 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.467829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.469106 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.469110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.469207 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.469230 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:35:51.469298 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 12:35:51.469365 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":288,"threads_started":1}
I20260619 12:35:51.470156 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.463420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37503 (local address 127.0.0.1:33338)
0619 12:35:51.463931 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:51.463944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:51.464034 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:51.465991 (+ 1957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:51.465994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:51.466011 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:51.466102 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.466108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.467396 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.467400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.468548 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.468558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.469771 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.469775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.469865 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.469889 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 12:35:51.469948 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 12:35:51.470018 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":105,"thread_start_us":88,"threads_started":1}
I20260619 12:35:51.470552 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.463726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48380 (local address 127.0.98.1:34107)
0619 12:35:51.464255 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.464267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.464288 (+ 21us) server_negotiation.cc:407] Connection header received
0619 12:35:51.464366 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.464371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.464538 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.464650 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:51.464846 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.465798 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.467956 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.468819 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.468910 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.469044 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.469692 (+ 648us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.470376 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":82,"threads_started":1}
I20260619 12:35:51.471020 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.464142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.0.98.2:37503)
0619 12:35:51.465576 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.465586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.465602 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:35:51.465686 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.465694 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.465861 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.466021 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:51.466180 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.467267 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.468835 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.469626 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.470581 (+ 955us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.470734 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.470786 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.470874 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1350,"thread_start_us":108,"threads_started":1}
I20260619 12:35:51.470772 7675 tablet_service.cc:1467] Processing CreateTablet for tablet ace6b4748e254b31b8fae39a988725e7 (DEFAULT_TABLE table=test-table [id=c5ea72a84fb148bd8fc0133693e12a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:51.471916 7675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace6b4748e254b31b8fae39a988725e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:51.473322 7539 tablet_service.cc:1467] Processing CreateTablet for tablet ace6b4748e254b31b8fae39a988725e7 (DEFAULT_TABLE table=test-table [id=c5ea72a84fb148bd8fc0133693e12a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:51.473366 7402 tablet_service.cc:1467] Processing CreateTablet for tablet ace6b4748e254b31b8fae39a988725e7 (DEFAULT_TABLE table=test-table [id=c5ea72a84fb148bd8fc0133693e12a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:51.474216 7539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace6b4748e254b31b8fae39a988725e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:51.474314 7402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace6b4748e254b31b8fae39a988725e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:51.480194 7775 tablet_bootstrap.cc:492] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a: Bootstrap starting.
I20260619 12:35:51.482065 7777 tablet_bootstrap.cc:492] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5: Bootstrap starting.
I20260619 12:35:51.482623 7776 tablet_bootstrap.cc:492] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6: Bootstrap starting.
I20260619 12:35:51.482805 7775 tablet_bootstrap.cc:654] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:51.483618 7775 log.cc:824] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:51.484229 7777 tablet_bootstrap.cc:654] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:51.484704 7776 tablet_bootstrap.cc:654] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:51.485108 7777 log.cc:824] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:51.485463 7776 log.cc:824] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:51.486187 7775 tablet_bootstrap.cc:492] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a: No bootstrap required, opened a new log
I20260619 12:35:51.486382 7775 ts_tablet_manager.cc:1397] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:35:51.486902 7777 tablet_bootstrap.cc:492] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5: No bootstrap required, opened a new log
I20260619 12:35:51.487082 7777 ts_tablet_manager.cc:1397] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:35:51.489271 7776 tablet_bootstrap.cc:492] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6: No bootstrap required, opened a new log
I20260619 12:35:51.489475 7776 ts_tablet_manager.cc:1397] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260619 12:35:51.490047 7777 raft_consensus.cc:348] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:51.490274 7777 raft_consensus.cc:374] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:51.490325 7777 raft_consensus.cc:729] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 114c34026d3c4740a1794ba290c620f5, State: Initialized, Role: FOLLOWER
I20260619 12:35:51.490758 7777 consensus_queue.cc:260] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:51.490628 7775 raft_consensus.cc:348] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:51.490890 7775 raft_consensus.cc:374] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:51.490957 7775 raft_consensus.cc:729] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c23456507c944441a0bbe0b9d3cc6e5a, State: Initialized, Role: FOLLOWER
I20260619 12:35:51.491420 7775 consensus_queue.cc:260] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:51.491829 7777 ts_tablet_manager.cc:1428] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 12:35:51.491997 7469 heartbeater.cc:503] Master 127.0.98.62:43151 was elected leader, sending a full tablet report...
I20260619 12:35:51.492367 7775 ts_tablet_manager.cc:1428] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:35:51.494287 7742 heartbeater.cc:503] Master 127.0.98.62:43151 was elected leader, sending a full tablet report...
I20260619 12:35:51.495064 7776 raft_consensus.cc:348] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:51.495334 7776 raft_consensus.cc:374] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:51.495384 7776 raft_consensus.cc:729] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bb8dd6c140400b91cceefd251251c6, State: Initialized, Role: FOLLOWER
I20260619 12:35:51.495841 7776 consensus_queue.cc:260] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:51.496682 7606 heartbeater.cc:503] Master 127.0.98.62:43151 was elected leader, sending a full tablet report...
I20260619 12:35:51.496836 7776 ts_tablet_manager.cc:1428] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 12:35:51.497583 7781 raft_consensus.cc:482] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:51.497732 7781 raft_consensus.cc:504] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:51.499310 7781 leader_election.cc:288] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c23456507c944441a0bbe0b9d3cc6e5a (127.0.98.3:33795), f5bb8dd6c140400b91cceefd251251c6 (127.0.98.2:37503)
I20260619 12:35:51.503762 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.499693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45915 (local address 127.0.98.3:33795)
0619 12:35:51.499803 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.499810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.499824 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:35:51.499867 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.499870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.499919 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.499991 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:51.500243 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.500934 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.502617 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.503183 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.503302 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.503361 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.503566 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.503642 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 12:35:51.503770 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.499113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33795 (local address 127.0.98.1:45915)
0619 12:35:51.499220 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:51.499235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:51.499314 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:51.499993 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:51.499996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:51.500009 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:51.500127 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.500134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.501063 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.501066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.502209 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.502221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.503303 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.503306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.503415 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.503437 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:35:51.503549 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 12:35:51.503625 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 12:35:51.504113 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.499607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37503 (local address 127.0.98.1:46509)
0619 12:35:51.499769 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:51.499781 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:51.499862 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:51.500181 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:51.500183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:51.500199 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:51.500323 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.500330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.501224 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.501230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.502839 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:51.502848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.503719 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.503722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:51.503826 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.503853 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 12:35:51.503927 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 12:35:51.504001 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 12:35:51.504454 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.499642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46509 (local address 127.0.98.2:37503)
0619 12:35:51.499752 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.499756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.499826 (+ 70us) server_negotiation.cc:407] Connection header received
0619 12:35:51.499950 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.499952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.500006 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.500078 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:51.500444 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.501118 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.502999 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.503464 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.503608 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.503664 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.504279 (+ 615us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.504365 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 12:35:51.504843 7695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace6b4748e254b31b8fae39a988725e7" candidate_uuid: "114c34026d3c4740a1794ba290c620f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" is_pre_election: true
I20260619 12:35:51.504913 7559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace6b4748e254b31b8fae39a988725e7" candidate_uuid: "114c34026d3c4740a1794ba290c620f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bb8dd6c140400b91cceefd251251c6" is_pre_election: true
I20260619 12:35:51.505142 7695 raft_consensus.cc:2454] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 114c34026d3c4740a1794ba290c620f5 in term 0.
I20260619 12:35:51.505159 7559 raft_consensus.cc:2454] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 114c34026d3c4740a1794ba290c620f5 in term 0.
I20260619 12:35:51.505573 7358 leader_election.cc:302] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 114c34026d3c4740a1794ba290c620f5, c23456507c944441a0bbe0b9d3cc6e5a; no voters:
I20260619 12:35:51.505821 7781 raft_consensus.cc:2781] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:51.505908 7781 raft_consensus.cc:482] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:51.505954 7781 raft_consensus.cc:3037] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:51.507014 7781 raft_consensus.cc:504] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:51.507347 7781 leader_election.cc:288] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [CANDIDATE]: Term 1 election: Requested vote from peers c23456507c944441a0bbe0b9d3cc6e5a (127.0.98.3:33795), f5bb8dd6c140400b91cceefd251251c6 (127.0.98.2:37503)
I20260619 12:35:51.507699 7695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace6b4748e254b31b8fae39a988725e7" candidate_uuid: "114c34026d3c4740a1794ba290c620f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c23456507c944441a0bbe0b9d3cc6e5a"
I20260619 12:35:51.507748 7559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace6b4748e254b31b8fae39a988725e7" candidate_uuid: "114c34026d3c4740a1794ba290c620f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bb8dd6c140400b91cceefd251251c6"
I20260619 12:35:51.507833 7695 raft_consensus.cc:3037] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:51.507845 7559 raft_consensus.cc:3037] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:51.509131 7695 raft_consensus.cc:2454] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 114c34026d3c4740a1794ba290c620f5 in term 1.
I20260619 12:35:51.509131 7559 raft_consensus.cc:2454] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 114c34026d3c4740a1794ba290c620f5 in term 1.
I20260619 12:35:51.509397 7358 leader_election.cc:302] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 114c34026d3c4740a1794ba290c620f5, f5bb8dd6c140400b91cceefd251251c6; no voters:
I20260619 12:35:51.509554 7781 raft_consensus.cc:2781] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:51.509775 7781 raft_consensus.cc:686] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [term 1 LEADER]: Becoming Leader. State: Replica: 114c34026d3c4740a1794ba290c620f5, State: Running, Role: LEADER
I20260619 12:35:51.510123 7781 consensus_queue.cc:237] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:51.513653 6505 catalog_manager.cc:5495] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 114c34026d3c4740a1794ba290c620f5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "114c34026d3c4740a1794ba290c620f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } health_report { overall_health: UNKNOWN } } }
W20260619 12:35:51.514571 7471 tablet.cc:2369] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:35:51.531667 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.528088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.0.98.1:34107)
0619 12:35:51.528224 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.528229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.528243 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:35:51.528399 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.528409 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.528471 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.528579 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:51.528807 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.529312 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.530171 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.530396 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.530518 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.530818 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:51.531249 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:51.531383 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.531430 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.531520 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:35:51.538297 7559 raft_consensus.cc:1261] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6 [term 1 FOLLOWER]: Refusing update from remote peer 114c34026d3c4740a1794ba290c620f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:51.538304 7695 raft_consensus.cc:1261] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a [term 1 FOLLOWER]: Refusing update from remote peer 114c34026d3c4740a1794ba290c620f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:51.539067 7781 consensus_queue.cc:1035] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:51.539333 7784 consensus_queue.cc:1035] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:51.548765 7789 mvcc.cc:205] Tried to move back new op lower bound from 7298549971092733952 to 7298549970988576768. Current Snapshot: MvccSnapshot[applied={T|T < 7298549971092733952}]
I20260619 12:35:51.549739 7788 mvcc.cc:205] Tried to move back new op lower bound from 7298549971092733952 to 7298549970988576768. Current Snapshot: MvccSnapshot[applied={T|T < 7298549971092733952}]
I20260619 12:35:51.557379 6506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:51.557572 6506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:51.561879 6506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a6c5cff1ee44943a5f87b30f25d9083: Sending DeleteTablet for 3 replicas of tablet ace6b4748e254b31b8fae39a988725e7
I20260619 12:35:51.562651 7402 tablet_service.cc:1514] Processing DeleteTablet for tablet ace6b4748e254b31b8fae39a988725e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48380
I20260619 12:35:51.562875 7675 tablet_service.cc:1514] Processing DeleteTablet for tablet ace6b4748e254b31b8fae39a988725e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52408
I20260619 12:35:51.563422 7539 tablet_service.cc:1514] Processing DeleteTablet for tablet ace6b4748e254b31b8fae39a988725e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33338
I20260619 12:35:51.563560 7798 tablet_replica.cc:330] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a: stopping tablet replica
I20260619 12:35:51.563966 7797 tablet_replica.cc:330] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5: stopping tablet replica
I20260619 12:35:51.564000 7798 raft_consensus.cc:2229] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:51.564252 7797 raft_consensus.cc:2229] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:35:51.564687 7797 raft_consensus.cc:2258] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:51.564812 7798 raft_consensus.cc:2258] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:51.566215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 6474
I20260619 12:35:51.567296 7798 ts_tablet_manager.cc:1905] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:51.570238 7798 ts_tablet_manager.cc:1918] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:51.570425 7798 log.cc:1196] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/ace6b4748e254b31b8fae39a988725e7
I20260619 12:35:51.570812 7798 ts_tablet_manager.cc:1939] T ace6b4748e254b31b8fae39a988725e7 P c23456507c944441a0bbe0b9d3cc6e5a: Deleting consensus metadata
I20260619 12:35:51.572006 7799 tablet_replica.cc:330] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6: stopping tablet replica
I20260619 12:35:51.572350 7799 raft_consensus.cc:2229] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:51.572710 7799 raft_consensus.cc:2258] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:51.574597 7799 ts_tablet_manager.cc:1905] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:51.577378 7799 ts_tablet_manager.cc:1918] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:51.577474 7799 log.cc:1196] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/ace6b4748e254b31b8fae39a988725e7
I20260619 12:35:51.577795 7799 ts_tablet_manager.cc:1939] T ace6b4748e254b31b8fae39a988725e7 P f5bb8dd6c140400b91cceefd251251c6: Deleting consensus metadata
I20260619 12:35:51.579118 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.565640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45246 (local address 127.0.98.61:40045)
0619 12:35:51.565750 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.565754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.567299 (+ 1545us) server_negotiation.cc:407] Connection header received
0619 12:35:51.567500 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.567504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.567549 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.567599 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:51.568662 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.569858 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.571059 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.571411 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.571505 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.574814 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:51.574818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:51.574821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:51.574850 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:51.576804 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.577396 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.577403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.577408 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.577499 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.578044 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.578051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.578055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.578336 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:51.578455 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.578842 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.578968 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 12:35:51.579636 7495 connection.cc:535] server connection from 127.0.0.1:33338 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 12:35:51.580207 7628 connection.cc:535] server connection from 127.0.0.1:52408 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 12:35:51.583241 7797 ts_tablet_manager.cc:1905] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:51.586230 7797 ts_tablet_manager.cc:1918] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:51.586345 7797 log.cc:1196] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-0/wal/wals/ace6b4748e254b31b8fae39a988725e7
I20260619 12:35:51.586665 7797 ts_tablet_manager.cc:1939] T ace6b4748e254b31b8fae39a988725e7 P 114c34026d3c4740a1794ba290c620f5: Deleting consensus metadata
W20260619 12:35:51.587805 7358 connection.cc:441] server connection from 127.0.0.1:48380 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48380 (request call id 1) could send its response
I20260619 12:35:51.590000 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.564515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.0.98.60:42811)
0619 12:35:51.564624 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.564629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.580834 (+ 16205us) server_negotiation.cc:407] Connection header received
0619 12:35:51.581030 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.581033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.581080 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.581126 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:51.582012 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.582800 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.584076 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.584444 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.584512 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.586348 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:51.586352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:51.586354 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:51.586380 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:51.588188 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.588726 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.588729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.588732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.588808 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.589069 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.589075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.589079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.589295 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:51.589391 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.589719 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.589845 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 12:35:51.600778 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.591357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45262 (local address 127.0.98.61:40045)
0619 12:35:51.591603 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.591608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.592020 (+ 412us) server_negotiation.cc:407] Connection header received
0619 12:35:51.592231 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.592234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.592285 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.592363 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:51.593247 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.593865 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.595001 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.595351 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.595456 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.597014 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:51.597018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:51.597021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:51.597052 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:51.598739 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.599331 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.599337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.599341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.599428 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.599882 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.599885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.599887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.600104 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:51.600232 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.600495 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.600615 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:35:51.600845 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:51.592034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56884 (local address 127.0.98.60:42811)
0619 12:35:51.592142 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:51.592146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:51.592410 (+ 264us) server_negotiation.cc:407] Connection header received
0619 12:35:51.592571 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:51.592574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:51.592612 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:51.592688 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:51.593437 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.594278 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.595239 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:51.595563 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:51.595655 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:51.597282 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:51.597285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:51.597288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:51.597315 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:51.598863 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.599593 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.599596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.599598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.599674 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:51.599989 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:51.599993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:51.599994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:51.600189 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:51.600303 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:51.600590 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 12:35:51.600698 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 12:35:52.552904 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:52.552459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.3:55311)
0619 12:35:52.552704 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:52.552808 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":88,"threads_started":1}
W20260619 12:35:52.553272 7742 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43151 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
I20260619 12:35:52.553992 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:52.553573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.1:59045)
0619 12:35:52.553834 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:52.553903 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":63,"threads_started":1}
I20260619 12:35:52.554219 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:52.553257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.2:59899)
0619 12:35:52.553790 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:52.553874 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":68,"threads_started":1}
W20260619 12:35:52.554262 7469 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43151 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
W20260619 12:35:52.555260 7606 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43151 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
I20260619 12:35:53.061930 7810 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5a6c5cff1ee44943a5f87b30f25d9083)
I20260619 12:35:53.062160 7810 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } }
I20260619 12:35:53.062671 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5a6c5cff1ee44943a5f87b30f25d9083)
I20260619 12:35:53.062862 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } }
I20260619 12:35:53.063359 7810 leader_election.cc:288] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a6c5cff1ee44943a5f87b30f25d9083 (127.0.98.62:43151), 9704aa957af5431b9e7527b8b3adbe0a (127.0.98.61:40045)
I20260619 12:35:53.064011 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.063451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.0.1:59274)
0619 12:35:53.063815 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.063908 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":71,"threads_started":1}
W20260619 12:35:53.064224 7276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
I20260619 12:35:53.064400 7811 leader_election.cc:288] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a6c5cff1ee44943a5f87b30f25d9083 (127.0.98.62:43151), ac343c85736a403ab6e799a52157d160 (127.0.98.60:42811)
I20260619 12:35:53.064622 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.064041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.0.1:59282)
0619 12:35:53.064441 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.064526 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":78,"threads_started":1}
W20260619 12:35:53.064801 7093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
I20260619 12:35:53.065572 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.065283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.0.1:59286)
0619 12:35:53.065424 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.065487 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 12:35:53.065707 7276 leader_election.cc:334] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5a6c5cff1ee44943a5f87b30f25d9083 (127.0.98.62:43151): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
I20260619 12:35:53.066138 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.065883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.0.1:59294)
0619 12:35:53.066035 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.066081 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 12:35:53.066334 7093 leader_election.cc:334] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5a6c5cff1ee44943a5f87b30f25d9083 (127.0.98.62:43151): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
I20260619 12:35:53.068390 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.063633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40045 (local address 127.0.0.1:45264)
0619 12:35:53.064135 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.064148 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:53.064295 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:53.064742 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:53.064747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:53.064767 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:53.064891 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:53.064900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.065930 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.065936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:53.067113 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:53.067120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.068079 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.068081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:53.068148 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:53.068168 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 12:35:53.068229 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 12:35:53.068285 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":139,"threads_started":1}
I20260619 12:35:53.068481 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.063736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45264 (local address 127.0.98.61:40045)
0619 12:35:53.064002 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:53.064007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:53.064216 (+ 209us) server_negotiation.cc:407] Connection header received
0619 12:35:53.064421 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:53.064427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:53.064483 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:53.064543 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:53.065028 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.065795 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.067232 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.067963 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.068086 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:53.068148 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:53.068249 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 12:35:53.068320 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":204,"thread_start_us":91,"threads_started":1}
I20260619 12:35:53.068876 7113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac343c85736a403ab6e799a52157d160" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9704aa957af5431b9e7527b8b3adbe0a" is_pre_election: true
I20260619 12:35:53.069095 7113 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac343c85736a403ab6e799a52157d160 in term 1.
I20260619 12:35:53.069458 7273 leader_election.cc:302] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9704aa957af5431b9e7527b8b3adbe0a, ac343c85736a403ab6e799a52157d160; no voters: 5a6c5cff1ee44943a5f87b30f25d9083
I20260619 12:35:53.069588 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.064094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42811 (local address 127.0.0.1:56886)
0619 12:35:53.064600 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.064614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:53.064716 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:53.065536 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:53.065538 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:53.065555 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:53.065630 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:53.065635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.066955 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.066958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:53.068358 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:53.068367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.069140 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.069143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:53.069258 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:53.069285 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 12:35:53.069374 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 12:35:53.069454 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":59,"thread_start_us":50,"threads_started":1}
I20260619 12:35:53.069685 7810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 12:35:53.069744 7810 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5a6c5cff1ee44943a5f87b30f25d9083)
I20260619 12:35:53.069787 7810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:35:53.069993 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.064569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.0.98.60:42811)
0619 12:35:53.064994 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:53.065006 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:53.065018 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:35:53.065052 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:53.065054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:53.065092 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:53.065446 (+ 354us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:53.065993 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.066827 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.068485 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.069034 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.069141 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:53.069197 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:53.069847 (+ 650us) server_negotiation.cc:299] Negotiation successful
0619 12:35:53.069908 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":355,"thread_start_us":79,"threads_started":1}
I20260619 12:35:53.070363 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9704aa957af5431b9e7527b8b3adbe0a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ac343c85736a403ab6e799a52157d160" is_pre_election: true
I20260619 12:35:53.070984 7810 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } }
I20260619 12:35:53.071269 7298 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9704aa957af5431b9e7527b8b3adbe0a in current term 2: Already voted for candidate ac343c85736a403ab6e799a52157d160 in this term.
I20260619 12:35:53.071442 7810 leader_election.cc:288] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [CANDIDATE]: Term 2 election: Requested vote from peers 5a6c5cff1ee44943a5f87b30f25d9083 (127.0.98.62:43151), 9704aa957af5431b9e7527b8b3adbe0a (127.0.98.61:40045)
I20260619 12:35:53.071781 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.071555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.0.1:59302)
0619 12:35:53.071646 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.071711 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 12:35:53.071900 7091 leader_election.cc:302] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9704aa957af5431b9e7527b8b3adbe0a; no voters: 5a6c5cff1ee44943a5f87b30f25d9083, ac343c85736a403ab6e799a52157d160
I20260619 12:35:53.072104 7811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:35:53.072317 7113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac343c85736a403ab6e799a52157d160" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9704aa957af5431b9e7527b8b3adbe0a"
I20260619 12:35:53.072479 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.072202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.0.1:59314)
0619 12:35:53.072330 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.072381 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 12:35:53.072602 7276 leader_election.cc:334] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5a6c5cff1ee44943a5f87b30f25d9083 (127.0.98.62:43151): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
I20260619 12:35:53.073220 7811 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:35:53.074204 7113 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac343c85736a403ab6e799a52157d160 in term 2.
I20260619 12:35:53.074762 7273 leader_election.cc:302] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9704aa957af5431b9e7527b8b3adbe0a, ac343c85736a403ab6e799a52157d160; no voters: 5a6c5cff1ee44943a5f87b30f25d9083
I20260619 12:35:53.074929 7810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 12:35:53.075153 7810 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [term 2 LEADER]: Becoming Leader. State: Replica: ac343c85736a403ab6e799a52157d160, State: Running, Role: LEADER
I20260619 12:35:53.075416 7810 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } }
I20260619 12:35:53.076824 7820 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac343c85736a403ab6e799a52157d160. Latest consensus state: current_term: 2 leader_uuid: "ac343c85736a403ab6e799a52157d160" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } } }
I20260619 12:35:53.076959 7820 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:53.077234 7822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:53.079377 7822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:53.079824 7822 catalog_manager.cc:1259] Loaded cluster ID: 88dd114f19f84679bc91e6d4d7d7c049
I20260619 12:35:53.079890 7822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:53.080444 7822 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:53.080861 7822 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: Loaded TSK: 0
I20260619 12:35:53.081606 7822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:35:53.125345 7288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:35:53.128527 7288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:53.131793 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.131511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.0.1:59336)
0619 12:35:53.131634 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.131707 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 12:35:53.131929 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [term 2 FOLLOWER]: Refusing update from remote peer ac343c85736a403ab6e799a52157d160: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 12:35:53.132457 7820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 12:35:53.132546 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.132318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.0.1:59340)
0619 12:35:53.132429 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.132490 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 12:35:53.132680 7276 consensus_peers.cc:487] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 -> Peer 5a6c5cff1ee44943a5f87b30f25d9083 (127.0.98.62:43151): Couldn't send request to peer 5a6c5cff1ee44943a5f87b30f25d9083. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 12:35:53.133975 7811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac343c85736a403ab6e799a52157d160. Latest consensus state: current_term: 2 leader_uuid: "ac343c85736a403ab6e799a52157d160" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } } }
I20260619 12:35:53.134112 7811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:35:53.135111 7820 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ac343c85736a403ab6e799a52157d160" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } } }
I20260619 12:35:53.135262 7820 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:53.135838 7811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ac343c85736a403ab6e799a52157d160" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a6c5cff1ee44943a5f87b30f25d9083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } peers { permanent_uuid: "9704aa957af5431b9e7527b8b3adbe0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40045 } attrs { promote: false } } peers { permanent_uuid: "ac343c85736a403ab6e799a52157d160" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42811 } attrs { promote: false } } }
I20260619 12:35:53.135965 7811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9704aa957af5431b9e7527b8b3adbe0a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:35:53.136010 7824 mvcc.cc:205] Tried to move back new op lower bound from 7298549977622155264 to 7298549977399422976. Current Snapshot: MvccSnapshot[applied={T|T < 7298549977622155264}]
I20260619 12:35:53.145691 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.141157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33795 (local address 127.0.0.1:52424)
0619 12:35:53.141304 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.141319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:53.141413 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:53.141905 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:53.141908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:53.141925 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:53.142026 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:53.142032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.143002 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.143006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:53.144322 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:53.144331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.145231 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.145237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:53.145334 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:53.145386 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 12:35:53.145453 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 12:35:53.145544 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":24}
I20260619 12:35:53.145843 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.140931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34107 (local address 127.0.0.1:48396)
0619 12:35:53.141272 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.141289 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:53.141376 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:53.142336 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:53.142339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:53.142358 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:53.142437 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:53.142443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.143282 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.143285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:53.144455 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:53.144463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.145347 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.145351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:53.145476 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:53.145501 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 12:35:53.145603 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 12:35:53.145671 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":31}
I20260619 12:35:53.145963 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.141100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.0.98.1:34107)
0619 12:35:53.142086 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:53.142092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:53.142107 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:35:53.142162 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:53.142166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:53.142213 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:53.142315 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:53.142565 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.143159 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.144603 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.145223 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.145398 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:53.145456 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:53.145621 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 12:35:53.145694 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":70,"threads_started":1}
I20260619 12:35:53.146476 7402 tablet_service.cc:1467] Processing CreateTablet for tablet f539bf7feb124ee1a7b0ce962ac41f8a (DEFAULT_TABLE table=test-table [id=2453e19b411a47f5bdee66a6de86c18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:53.146538 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.141399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.0.98.3:33795)
0619 12:35:53.141641 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:53.141645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:53.141656 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:35:53.141700 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:53.141702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:53.141764 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:53.141855 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:53.142142 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.142878 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.144492 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.145107 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.145223 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:53.145316 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:53.146316 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0619 12:35:53.146406 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
I20260619 12:35:53.146800 7402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f539bf7feb124ee1a7b0ce962ac41f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:53.147037 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.141707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37503 (local address 127.0.0.1:33352)
0619 12:35:53.142227 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.142243 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:53.142322 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:53.143294 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:53.143296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:53.143314 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:53.143434 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:53.143442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.144279 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.144282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:53.145185 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:53.145198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.146473 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.146476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:53.146589 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:53.146618 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 12:35:53.146835 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 12:35:53.146904 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":144,"threads_started":1}
I20260619 12:35:53.147058 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.141821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33352 (local address 127.0.98.2:37503)
0619 12:35:53.142064 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:53.142068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:53.142545 (+ 477us) server_negotiation.cc:407] Connection header received
0619 12:35:53.142635 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:53.142638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:53.143066 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:53.143151 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:53.143564 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.144157 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.145337 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.145961 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.146364 (+ 403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:53.146426 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:53.146853 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 12:35:53.146931 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
I20260619 12:35:53.148057 7675 tablet_service.cc:1467] Processing CreateTablet for tablet f539bf7feb124ee1a7b0ce962ac41f8a (DEFAULT_TABLE table=test-table [id=2453e19b411a47f5bdee66a6de86c18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:53.148350 7675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f539bf7feb124ee1a7b0ce962ac41f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:53.149559 7832 tablet_bootstrap.cc:492] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5: Bootstrap starting.
I20260619 12:35:53.149744 7539 tablet_service.cc:1467] Processing CreateTablet for tablet f539bf7feb124ee1a7b0ce962ac41f8a (DEFAULT_TABLE table=test-table [id=2453e19b411a47f5bdee66a6de86c18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:53.150024 7539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f539bf7feb124ee1a7b0ce962ac41f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:53.150548 7832 tablet_bootstrap.cc:654] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:53.150856 7833 tablet_bootstrap.cc:492] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a: Bootstrap starting.
I20260619 12:35:53.151823 7833 tablet_bootstrap.cc:654] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:53.152529 7836 tablet_bootstrap.cc:492] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6: Bootstrap starting.
I20260619 12:35:53.152977 7833 tablet_bootstrap.cc:492] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a: No bootstrap required, opened a new log
I20260619 12:35:53.153039 7833 ts_tablet_manager.cc:1397] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 12:35:53.153252 7832 tablet_bootstrap.cc:492] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5: No bootstrap required, opened a new log
I20260619 12:35:53.153334 7832 ts_tablet_manager.cc:1397] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 12:35:53.153364 7833 raft_consensus.cc:348] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:53.153447 7833 raft_consensus.cc:374] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:53.153468 7833 raft_consensus.cc:729] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c23456507c944441a0bbe0b9d3cc6e5a, State: Initialized, Role: FOLLOWER
I20260619 12:35:53.153499 7836 tablet_bootstrap.cc:654] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:53.153610 7833 consensus_queue.cc:260] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:53.153784 7832 raft_consensus.cc:348] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:53.153906 7832 raft_consensus.cc:374] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:53.153944 7832 raft_consensus.cc:729] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 114c34026d3c4740a1794ba290c620f5, State: Initialized, Role: FOLLOWER
I20260619 12:35:53.153944 7833 ts_tablet_manager.cc:1428] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:35:53.154259 7832 consensus_queue.cc:260] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:53.154642 7832 ts_tablet_manager.cc:1428] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 12:35:53.155611 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.154414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.3:41933)
0619 12:35:53.155384 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.155452 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":893,"thread_start_us":83,"threads_started":1}
I20260619 12:35:53.156380 7741 heartbeater.cc:503] Master 127.0.98.60:42811 was elected leader, sending a full tablet report...
I20260619 12:35:53.156653 7468 heartbeater.cc:503] Master 127.0.98.60:42811 was elected leader, sending a full tablet report...
I20260619 12:35:53.156791 7836 tablet_bootstrap.cc:492] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6: No bootstrap required, opened a new log
I20260619 12:35:53.156872 7836 ts_tablet_manager.cc:1397] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 12:35:53.157344 7836 raft_consensus.cc:348] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:53.157472 7836 raft_consensus.cc:374] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:53.157514 7836 raft_consensus.cc:729] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bb8dd6c140400b91cceefd251251c6, State: Initialized, Role: FOLLOWER
I20260619 12:35:53.157685 7836 consensus_queue.cc:260] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:53.158059 7836 ts_tablet_manager.cc:1428] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:35:53.159133 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.156423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.3:44467)
0619 12:35:53.158964 (+ 2541us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.159039 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2473,"thread_start_us":92,"threads_started":1}
I20260619 12:35:53.159185 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.158336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.2:57101)
0619 12:35:53.159010 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.159070 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":90,"threads_started":1}
I20260619 12:35:53.159686 7605 heartbeater.cc:503] Master 127.0.98.60:42811 was elected leader, sending a full tablet report...
I20260619 12:35:53.159718 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.158421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.1:45847)
0619 12:35:53.159560 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.159618 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1073,"thread_start_us":100,"threads_started":1}
I20260619 12:35:53.159989 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.159594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.2:60525)
0619 12:35:53.159867 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.159917 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
I20260619 12:35:53.160481 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.160089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.1:34287)
0619 12:35:53.160361 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.160411 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":104,"threads_started":1}
W20260619 12:35:53.166982 7744 tablet.cc:2369] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:35:53.206336 7837 raft_consensus.cc:482] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:53.206480 7837 raft_consensus.cc:504] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:53.207610 7837 leader_election.cc:288] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 114c34026d3c4740a1794ba290c620f5 (127.0.98.1:34107), f5bb8dd6c140400b91cceefd251251c6 (127.0.98.2:37503)
I20260619 12:35:53.211206 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.207946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47755 (local address 127.0.98.2:37503)
0619 12:35:53.208074 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:53.208079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:53.208097 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:35:53.208236 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:53.208238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:53.208278 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:53.208364 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:53.208636 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.209142 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.210206 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.210597 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.210725 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:53.210792 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:53.211017 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 12:35:53.211089 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:35:53.211231 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.207843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37503 (local address 127.0.98.3:47755)
0619 12:35:53.207969 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.207987 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:53.208103 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:53.208379 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:53.208382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:53.208399 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:53.208523 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:53.208528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.209261 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.209265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:53.210079 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:53.210087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.210739 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.210742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:53.210837 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:53.210905 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 12:35:53.210988 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 12:35:53.211062 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":26}
I20260619 12:35:53.211304 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.207956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38801 (local address 127.0.98.1:34107)
0619 12:35:53.208101 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:53.208106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:53.208307 (+ 201us) server_negotiation.cc:407] Connection header received
0619 12:35:53.208465 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:53.208468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:53.208516 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:53.208598 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:53.208870 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.209347 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.210315 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.210715 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.210828 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:53.210908 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:53.211107 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 12:35:53.211179 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 12:35:53.211333 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.207850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34107 (local address 127.0.98.3:38801)
0619 12:35:53.208206 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.208221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:53.208359 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:53.208608 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:53.208611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:53.208626 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:53.208742 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:53.208747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.209460 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.209464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:53.210192 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:53.210200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.210832 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.210836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:53.210982 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:53.210999 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 12:35:53.211081 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 12:35:53.211153 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":289}
I20260619 12:35:53.211880 7559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f539bf7feb124ee1a7b0ce962ac41f8a" candidate_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bb8dd6c140400b91cceefd251251c6" is_pre_election: true
I20260619 12:35:53.211915 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f539bf7feb124ee1a7b0ce962ac41f8a" candidate_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "114c34026d3c4740a1794ba290c620f5" is_pre_election: true
I20260619 12:35:53.212021 7559 raft_consensus.cc:2454] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c23456507c944441a0bbe0b9d3cc6e5a in term 0.
I20260619 12:35:53.212186 7422 raft_consensus.cc:2454] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c23456507c944441a0bbe0b9d3cc6e5a in term 0.
I20260619 12:35:53.212428 7631 leader_election.cc:302] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c23456507c944441a0bbe0b9d3cc6e5a, f5bb8dd6c140400b91cceefd251251c6; no voters:
I20260619 12:35:53.212728 7837 raft_consensus.cc:2781] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:53.212782 7837 raft_consensus.cc:482] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:53.212810 7837 raft_consensus.cc:3037] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:53.213863 7837 raft_consensus.cc:504] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:53.214222 7837 leader_election.cc:288] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [CANDIDATE]: Term 1 election: Requested vote from peers 114c34026d3c4740a1794ba290c620f5 (127.0.98.1:34107), f5bb8dd6c140400b91cceefd251251c6 (127.0.98.2:37503)
I20260619 12:35:53.214602 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f539bf7feb124ee1a7b0ce962ac41f8a" candidate_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "114c34026d3c4740a1794ba290c620f5"
I20260619 12:35:53.214588 7559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f539bf7feb124ee1a7b0ce962ac41f8a" candidate_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bb8dd6c140400b91cceefd251251c6"
I20260619 12:35:53.214715 7422 raft_consensus.cc:3037] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:53.214747 7559 raft_consensus.cc:3037] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:53.215896 7422 raft_consensus.cc:2454] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c23456507c944441a0bbe0b9d3cc6e5a in term 1.
I20260619 12:35:53.215896 7559 raft_consensus.cc:2454] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c23456507c944441a0bbe0b9d3cc6e5a in term 1.
I20260619 12:35:53.216200 7630 leader_election.cc:302] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 114c34026d3c4740a1794ba290c620f5, c23456507c944441a0bbe0b9d3cc6e5a; no voters:
I20260619 12:35:53.216377 7837 raft_consensus.cc:2781] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:53.216596 7837 raft_consensus.cc:686] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [term 1 LEADER]: Becoming Leader. State: Replica: c23456507c944441a0bbe0b9d3cc6e5a, State: Running, Role: LEADER
I20260619 12:35:53.217072 7837 consensus_queue.cc:237] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } }
I20260619 12:35:53.217474 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.217255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.3:33663)
0619 12:35:53.217356 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.217415 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 12:35:53.219599 7288 catalog_manager.cc:5495] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a reported cstate change: term changed from 0 to 1, leader changed from <none> to c23456507c944441a0bbe0b9d3cc6e5a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c23456507c944441a0bbe0b9d3cc6e5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 } health_report { overall_health: UNKNOWN } } }
W20260619 12:35:53.219825 7608 tablet.cc:2369] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:35:53.249270 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.245585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.0.98.3:33795)
0619 12:35:53.245717 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:53.245745 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:53.245761 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:35:53.245864 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:53.245867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:53.245921 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:53.246035 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:53.246287 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.246809 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.247756 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:53.247978 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:53.248091 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:53.248374 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:53.248854 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:53.248985 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:53.249052 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 12:35:53.249112 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 12:35:53.252386 7559 raft_consensus.cc:1261] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6 [term 1 FOLLOWER]: Refusing update from remote peer c23456507c944441a0bbe0b9d3cc6e5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:53.252612 7422 raft_consensus.cc:1261] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5 [term 1 FOLLOWER]: Refusing update from remote peer c23456507c944441a0bbe0b9d3cc6e5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:53.253026 7837 consensus_queue.cc:1035] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bb8dd6c140400b91cceefd251251c6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:53.253266 7847 consensus_queue.cc:1035] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "114c34026d3c4740a1794ba290c620f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:53.255718 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.255039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.2:50487)
0619 12:35:53.255537 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.255609 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386}
I20260619 12:35:53.256062 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.255340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.1:59685)
0619 12:35:53.255906 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.255976 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496}
I20260619 12:35:53.259097 7852 mvcc.cc:205] Tried to move back new op lower bound from 7298549978116161536 to 7298549977979809792. Current Snapshot: MvccSnapshot[applied={T|T < 7298549978116161536}]
I20260619 12:35:53.259228 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.258941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.3:40769)
0619 12:35:53.259097 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.259153 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 12:35:53.260154 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.259875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.1:44971)
0619 12:35:53.260022 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.260082 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 12:35:53.260775 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.260541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.2:36155)
0619 12:35:53.260642 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.260704 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 12:35:53.262790 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.262552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43151 (local address 127.0.98.3:52915)
0619 12:35:53.262674 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.262725 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 12:35:53.271252 7288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:53.271507 7288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:53.276019 7288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac343c85736a403ab6e799a52157d160: Sending DeleteTablet for 3 replicas of tablet f539bf7feb124ee1a7b0ce962ac41f8a
I20260619 12:35:53.276706 7402 tablet_service.cc:1514] Processing DeleteTablet for tablet f539bf7feb124ee1a7b0ce962ac41f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:53 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48396
I20260619 12:35:53.276969 7675 tablet_service.cc:1514] Processing DeleteTablet for tablet f539bf7feb124ee1a7b0ce962ac41f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:53 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:52424
I20260619 12:35:53.277283 7860 tablet_replica.cc:330] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5: stopping tablet replica
I20260619 12:35:53.277431 7860 raft_consensus.cc:2229] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:53.277437 7539 tablet_service.cc:1514] Processing DeleteTablet for tablet f539bf7feb124ee1a7b0ce962ac41f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:53 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33352
I20260619 12:35:53.277575 7860 raft_consensus.cc:2258] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:53.277755 7862 tablet_replica.cc:330] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6: stopping tablet replica
I20260619 12:35:53.277884 7862 raft_consensus.cc:2229] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:53.278023 7862 raft_consensus.cc:2258] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:53.278417 7860 ts_tablet_manager.cc:1905] T f539bf7feb124ee1a7b0ce962ac41f8a P 114c34026d3c4740a1794ba290c620f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:53.278708 7862 ts_tablet_manager.cc:1905] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:53.278841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 7250
I20260619 12:35:53.281836 7862 ts_tablet_manager.cc:1918] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:53.281905 7862 log.cc:1196] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/f539bf7feb124ee1a7b0ce962ac41f8a
I20260619 12:35:53.282193 7862 ts_tablet_manager.cc:1939] T f539bf7feb124ee1a7b0ce962ac41f8a P f5bb8dd6c140400b91cceefd251251c6: Deleting consensus metadata
I20260619 12:35:53.282931 7276 catalog_manager.cc:4841] TS f5bb8dd6c140400b91cceefd251251c6 (127.0.98.2:37503): tablet f539bf7feb124ee1a7b0ce962ac41f8a (table test-table [id=2453e19b411a47f5bdee66a6de86c18c]) successfully deleted
I20260619 12:35:53.288879 7861 tablet_replica.cc:330] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a: stopping tablet replica
I20260619 12:35:53.289005 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:53.288706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34107 (local address 127.0.0.1:48400)
0619 12:35:53.288854 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:53.288911 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 12:35:53.289055 7861 raft_consensus.cc:2229] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [term 1 LEADER]: Raft consensus shutting down.
W20260619 12:35:53.289202 7275 catalog_manager.cc:4584] TS 114c34026d3c4740a1794ba290c620f5 (127.0.98.1:34107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f539bf7feb124ee1a7b0ce962ac41f8a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34107: connect: Connection refused (error 111)
I20260619 12:35:53.289479 7861 raft_consensus.cc:2258] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:53.289669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 7476
I20260619 12:35:53.291008 7861 ts_tablet_manager.cc:1905] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:53.293496 7861 ts_tablet_manager.cc:1918] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:53.293602 7861 log.cc:1196] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/f539bf7feb124ee1a7b0ce962ac41f8a
I20260619 12:35:53.293886 7861 ts_tablet_manager.cc:1939] T f539bf7feb124ee1a7b0ce962ac41f8a P c23456507c944441a0bbe0b9d3cc6e5a: Deleting consensus metadata
I20260619 12:35:53.294646 7276 catalog_manager.cc:4841] TS c23456507c944441a0bbe0b9d3cc6e5a (127.0.98.3:33795): tablet f539bf7feb124ee1a7b0ce962ac41f8a (table test-table [id=2453e19b411a47f5bdee66a6de86c18c]) successfully deleted
I20260619 12:35:53.298157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 7612
I20260619 12:35:53.304813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 6970
I20260619 12:35:53.310714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 7077
2026-06-19T12:35:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5254 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:35:53 dist-test-slave-kdrb krb5kdc[7865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:35:53 dist-test-slave-kdrb krb5kdc[7865](info): set up 2 sockets
Jun 19 12:35:53 dist-test-slave-kdrb krb5kdc[7865](info): commencing operation
krb5kdc: starting...
W20260619 12:35:54.352144 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:35:54 dist-test-slave-kdrb krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:35:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 12:35:54.478955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42445
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:54.595201 7878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:54.595767 7878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:54.595861 7878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:54.600260 7878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:54.600338 7878 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 12:35:54.600356 7878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:54.600374 7878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:54.600391 7878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:54.600435 7878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:54.606391 7878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42445
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:54.607781 7878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:54.609005 7878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:54.617888 7883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:54.618026 7878 server_base.cc:1034] running on GCE node
W20260619 12:35:54.617930 7886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:54.618274 7884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:54.618961 7878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:54.620255 7878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:54.621433 7878 hybrid_clock.cc:648] HybridClock initialized: now 1781872554621420 us; error 32 us; skew 500 ppm
Jun 19 12:35:54 dist-test-slave-kdrb krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872554, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:54.624436 7878 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 12:35:54.625411 7878 webserver.cc:460] Webserver started at http://127.0.98.62:38545/ using document root <none> and password file <none>
I20260619 12:35:54.625972 7878 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:54.626031 7878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:54.626205 7878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:54.628414 7878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "d343ffe1e6f74f0bb0db9998777ef359"
format_stamp: "Formatted at 2026-06-19 12:35:54 on dist-test-slave-kdrb"
server_key: "e4dae51e4ec9faff52e886463762288f"
server_key_iv: "35766b17cea8253395813184132ce2e3"
server_key_version: "encryptionkey@0"
I20260619 12:35:54.628968 7878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "d343ffe1e6f74f0bb0db9998777ef359"
format_stamp: "Formatted at 2026-06-19 12:35:54 on dist-test-slave-kdrb"
server_key: "e4dae51e4ec9faff52e886463762288f"
server_key_iv: "35766b17cea8253395813184132ce2e3"
server_key_version: "encryptionkey@0"
I20260619 12:35:54.633244 7878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:35:54.635928 7893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:54.637290 7878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 12:35:54.637429 7878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "d343ffe1e6f74f0bb0db9998777ef359"
format_stamp: "Formatted at 2026-06-19 12:35:54 on dist-test-slave-kdrb"
server_key: "e4dae51e4ec9faff52e886463762288f"
server_key_iv: "35766b17cea8253395813184132ce2e3"
server_key_version: "encryptionkey@0"
I20260619 12:35:54.637547 7878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:54.660913 7878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:54.664654 7878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:54.664894 7878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:54.674409 7878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42445
I20260619 12:35:54.674415 7955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42445 every 8 connection(s)
I20260619 12:35:54.675756 7878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:54.676216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 7878
I20260619 12:35:54.676393 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:54.676756 392 external_mini_cluster.cc:1428] Setting key cef0cf3464e3d0d578c2ac6c1d4802a5
I20260619 12:35:54.679738 7956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 12:35:54 dist-test-slave-kdrb krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:35:54.687235 7956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359: Bootstrap starting.
I20260619 12:35:54.689977 7956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:54.691501 7956 log.cc:824] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:54.693449 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:54.678221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.0.98.62:42445)
0619 12:35:54.678663 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:54.678683 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:54.678726 (+ 43us) server_negotiation.cc:407] Connection header received
0619 12:35:54.679635 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:54.679661 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:54.680129 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:54.680523 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:54.681236 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:54.682794 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:54.683757 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:54.684481 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:54.684661 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:54.687634 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:54.687641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:54.687664 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:54.687702 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:54.689675 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:54.690283 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:54.690292 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:54.690303 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:54.690482 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:54.690790 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:54.690794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:54.690796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:54.691454 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:54.691687 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:54.692019 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 12:35:54.692399 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":129,"threads_started":1}
I20260619 12:35:54.694074 7956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359: No bootstrap required, opened a new log
I20260619 12:35:54.697579 7956 raft_consensus.cc:348] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d343ffe1e6f74f0bb0db9998777ef359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42445 } }
I20260619 12:35:54.697903 7956 raft_consensus.cc:374] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:54.697989 7956 raft_consensus.cc:729] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d343ffe1e6f74f0bb0db9998777ef359, State: Initialized, Role: FOLLOWER
I20260619 12:35:54.698465 7956 consensus_queue.cc:260] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d343ffe1e6f74f0bb0db9998777ef359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42445 } }
I20260619 12:35:54.698617 7956 raft_consensus.cc:388] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:54.698684 7956 raft_consensus.cc:482] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:54.698784 7956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:54.699889 7956 raft_consensus.cc:504] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d343ffe1e6f74f0bb0db9998777ef359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42445 } }
I20260619 12:35:54.700263 7956 leader_election.cc:302] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d343ffe1e6f74f0bb0db9998777ef359; no voters:
I20260619 12:35:54.700596 7956 leader_election.cc:288] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:54.700757 7961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:54.701012 7961 raft_consensus.cc:686] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [term 1 LEADER]: Becoming Leader. State: Replica: d343ffe1e6f74f0bb0db9998777ef359, State: Running, Role: LEADER
I20260619 12:35:54.701339 7961 consensus_queue.cc:237] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d343ffe1e6f74f0bb0db9998777ef359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42445 } }
I20260619 12:35:54.701936 7956 sys_catalog.cc:564] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:54.703245 7963 sys_catalog.cc:455] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d343ffe1e6f74f0bb0db9998777ef359. Latest consensus state: current_term: 1 leader_uuid: "d343ffe1e6f74f0bb0db9998777ef359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d343ffe1e6f74f0bb0db9998777ef359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42445 } } }
I20260619 12:35:54.703393 7963 sys_catalog.cc:458] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:54.703241 7962 sys_catalog.cc:455] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d343ffe1e6f74f0bb0db9998777ef359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d343ffe1e6f74f0bb0db9998777ef359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42445 } } }
I20260619 12:35:54.703447 7962 sys_catalog.cc:458] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359 [sys.catalog]: This master's current role is: LEADER
W20260619 12:35:54.707005 7976 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:35:54.707113 7976 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:35:54.707185 7971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:54.708073 7971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:54.713752 7971 catalog_manager.cc:1347] Generated new cluster ID: 1e6745d8fc364044a7a4069fbfe22730
I20260619 12:35:54.713873 7971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:54.736689 7971 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:54.737709 7971 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:54.749790 7971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359: Generated new TSK 0
I20260619 12:35:54.750656 7971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 12:35:54.786680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42445
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:54.902792 7982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:54.903091 7982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:54.903198 7982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:54.907518 7982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:54.907598 7982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:54.907640 7982 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:35:54.907675 7982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:35:54.913254 7982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:54.914338 7982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:54.915354 7982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:54.925531 7987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:54.925493 7990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:54.925642 7982 server_base.cc:1034] running on GCE node
W20260619 12:35:54.925879 7988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:54.926337 7982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:54.927083 7982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:54.928294 7982 hybrid_clock.cc:648] HybridClock initialized: now 1781872554928254 us; error 56 us; skew 500 ppm
Jun 19 12:35:54 dist-test-slave-kdrb krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872554, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:54.931224 7982 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 12:35:54.932302 7982 webserver.cc:460] Webserver started at http://127.0.98.1:44713/ using document root <none> and password file <none>
I20260619 12:35:54.932950 7982 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:54.933051 7982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:54.933293 7982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:54.935513 7982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "680b9950669e4d98abdc4b6ccf6fe0b4"
format_stamp: "Formatted at 2026-06-19 12:35:54 on dist-test-slave-kdrb"
server_key: "a6d97030080ae41767a784b926e9b70e"
server_key_iv: "47ce660e2932b73b9086534e838e71a7"
server_key_version: "encryptionkey@0"
I20260619 12:35:54.936151 7982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "680b9950669e4d98abdc4b6ccf6fe0b4"
format_stamp: "Formatted at 2026-06-19 12:35:54 on dist-test-slave-kdrb"
server_key: "a6d97030080ae41767a784b926e9b70e"
server_key_iv: "47ce660e2932b73b9086534e838e71a7"
server_key_version: "encryptionkey@0"
I20260619 12:35:54.940191 7982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:35:54.943172 7997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:54.944634 7982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 12:35:54.944787 7982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "680b9950669e4d98abdc4b6ccf6fe0b4"
format_stamp: "Formatted at 2026-06-19 12:35:54 on dist-test-slave-kdrb"
server_key: "a6d97030080ae41767a784b926e9b70e"
server_key_iv: "47ce660e2932b73b9086534e838e71a7"
server_key_version: "encryptionkey@0"
I20260619 12:35:54.944916 7982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:54.957239 7982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:54.960829 7982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:54.961056 7982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:54.962533 7982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:54.962611 7982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:54.962682 7982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:54.962714 7982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:35:54 dist-test-slave-kdrb krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872554, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:35:54.973883 7982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45309
I20260619 12:35:54.973969 8112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45309 every 8 connection(s)
I20260619 12:35:54.974944 7982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:35:54.976989 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:54.963551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47385 (local address 127.0.98.62:42445)
0619 12:35:54.963693 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:54.963697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:54.964760 (+ 1063us) server_negotiation.cc:407] Connection header received
0619 12:35:54.965937 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:54.965940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:54.966000 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:54.966096 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:54.967874 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:54.968427 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:54.969622 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:54.969931 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:54.970054 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:54.973052 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:54.973056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:54.973058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:54.973085 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:54.974798 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:54.975460 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:54.975464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:54.975465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:54.975582 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:54.975977 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:54.975980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:54.975981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:54.976173 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:54.976253 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:54.976752 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 12:35:54.976867 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:35:54.977837 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:54.964105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42445 (local address 127.0.98.1:47385)
0619 12:35:54.964601 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:54.964646 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:54.965684 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:54.966334 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:54.966347 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:54.966852 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:54.967678 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:54.967703 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:54.968569 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:54.968572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:54.969491 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:54.969500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:54.970112 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:54.970116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:54.970326 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:54.971046 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:54.971066 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:54.972847 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:54.974954 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:54.974963 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:54.974979 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:54.975326 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:54.975671 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:54.975676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:54.975678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:54.975853 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:54.976271 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:54.976276 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:54.976534 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 12:35:54.976688 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 12:35:54.976949 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":147,"threads_started":1}
I20260619 12:35:54.979321 8113 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42445
I20260619 12:35:54.979508 8113 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:54.980121 8113 heartbeater.cc:511] Master 127.0.98.62:42445 requested a full tablet report, sending...
I20260619 12:35:54.981993 7909 ts_manager.cc:194] Registered new tserver with Master: 680b9950669e4d98abdc4b6ccf6fe0b4 (127.0.98.1:45309)
I20260619 12:35:54.983597 7909 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47385
I20260619 12:35:54.983697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 7982
I20260619 12:35:54.983817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:35:54.984146 392 external_mini_cluster.cc:1428] Setting key 8cf35a1a2220ce3d4d8dae930cc39d24
I20260619 12:35:54.990255 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:54.981980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59713 (local address 127.0.98.62:42445)
0619 12:35:54.982129 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:54.982134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:54.982393 (+ 259us) server_negotiation.cc:407] Connection header received
0619 12:35:54.982491 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:54.982493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:54.982544 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:54.982611 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:54.983152 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:54.983857 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:54.985150 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:54.985476 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:54.985582 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:54.986848 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:54.986854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:54.986859 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:54.986890 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:54.988463 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:54.989005 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:54.989007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:54.989009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:54.989081 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:54.989430 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:54.989435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:54.989436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:54.989645 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:54.989725 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:54.990035 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 12:35:54.990135 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 12:35:54.990455 8119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:54.981907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42445 (local address 127.0.98.1:59713)
0619 12:35:54.982260 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:54.982289 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:54.982404 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:54.982656 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:54.982659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:54.982905 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:54.983038 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:54.983046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:54.984002 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:54.984006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:54.985022 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:54.985033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:54.985600 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:54.985604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:54.985693 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:54.986359 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:54.986371 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:54.986700 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:54.988586 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:54.988593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:54.988598 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:54.988896 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:54.989175 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:54.989181 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:54.989186 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:54.989308 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:54.989746 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:54.989749 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:54.989865 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 12:35:54.989987 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 12:35:54.990171 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":22,"thread_start_us":141,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 12:35:55.012583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42445
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:55.126767 8122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:55.127061 8122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:55.127107 8122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:55.131347 8122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:55.131426 8122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:55.131516 8122 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:35:55.131577 8122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:35:55.137202 8122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:55.138338 8122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:55.139400 8122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:55.149642 8127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:55.149642 8128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:55.149648 8130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:55.149860 8122 server_base.cc:1034] running on GCE node
I20260619 12:35:55.150429 8122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:55.151118 8122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:55.152349 8122 hybrid_clock.cc:648] HybridClock initialized: now 1781872555152311 us; error 52 us; skew 500 ppm
Jun 19 12:35:55 dist-test-slave-kdrb krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:55.155254 8122 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 12:35:55.156297 8122 webserver.cc:460] Webserver started at http://127.0.98.2:36849/ using document root <none> and password file <none>
I20260619 12:35:55.156899 8122 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:55.156994 8122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:55.157238 8122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:55.159411 8122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "9d57187001f6464b9a5721b92d3b102e"
format_stamp: "Formatted at 2026-06-19 12:35:55 on dist-test-slave-kdrb"
server_key: "fb97b9afeb9c46da18fbaa025e5a2abf"
server_key_iv: "861a16447ded23d64b23ffbb6dbd4b9e"
server_key_version: "encryptionkey@0"
I20260619 12:35:55.160094 8122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9d57187001f6464b9a5721b92d3b102e"
format_stamp: "Formatted at 2026-06-19 12:35:55 on dist-test-slave-kdrb"
server_key: "fb97b9afeb9c46da18fbaa025e5a2abf"
server_key_iv: "861a16447ded23d64b23ffbb6dbd4b9e"
server_key_version: "encryptionkey@0"
I20260619 12:35:55.164309 8122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:35:55.167001 8137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:55.168417 8122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:35:55.168573 8122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "9d57187001f6464b9a5721b92d3b102e"
format_stamp: "Formatted at 2026-06-19 12:35:55 on dist-test-slave-kdrb"
server_key: "fb97b9afeb9c46da18fbaa025e5a2abf"
server_key_iv: "861a16447ded23d64b23ffbb6dbd4b9e"
server_key_version: "encryptionkey@0"
I20260619 12:35:55.168702 8122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:55.181664 8122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:55.185092 8122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:55.185325 8122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:55.186909 8122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:55.186966 8122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:55.187036 8122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:55.187068 8122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:35:55 dist-test-slave-kdrb krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:35:55.199226 8122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37345
I20260619 12:35:55.199291 8252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37345 every 8 connection(s)
I20260619 12:35:55.200270 8122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:35:55.203372 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.187866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42449 (local address 127.0.98.62:42445)
0619 12:35:55.187994 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:55.188005 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:55.189333 (+ 1328us) server_negotiation.cc:407] Connection header received
0619 12:35:55.190587 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:55.190590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:55.190654 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:55.190746 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:55.192438 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.193352 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.195240 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.195643 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.195758 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.198959 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:55.198965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:55.198969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:55.199004 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:55.200952 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:55.201587 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:55.201590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:55.201591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:55.201676 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:55.202112 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:55.202116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:55.202117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:55.202340 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:55.202463 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:55.203024 (+ 561us) server_negotiation.cc:299] Negotiation successful
0619 12:35:55.203189 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:35:55.204341 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.188499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42445 (local address 127.0.98.2:42449)
0619 12:35:55.189129 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:55.189210 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:55.190291 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:55.190941 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:55.190950 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:55.191432 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:55.192216 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.192237 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.193523 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.193527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.195078 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.195090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.195945 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.195948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.196102 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.196782 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:55.196808 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:55.198752 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:55.201119 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:55.201125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:55.201141 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:55.201481 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:55.201775 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:55.201777 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:55.201779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:55.202008 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:55.202459 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:55.202466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:55.202736 (+ 270us) client_negotiation.cc:769] Sending connection context
0619 12:35:55.202966 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 12:35:55.203353 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":165,"threads_started":1}
I20260619 12:35:55.205735 8253 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42445
I20260619 12:35:55.205901 8253 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:55.206565 8253 heartbeater.cc:511] Master 127.0.98.62:42445 requested a full tablet report, sending...
I20260619 12:35:55.207882 7910 ts_manager.cc:194] Registered new tserver with Master: 9d57187001f6464b9a5721b92d3b102e (127.0.98.2:37345)
I20260619 12:35:55.208669 7910 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42449
I20260619 12:35:55.209761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 8122
I20260619 12:35:55.209895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:35:55.210177 392 external_mini_cluster.cc:1428] Setting key d1bd9385c1b66cf032d1802874700095
I20260619 12:35:55.216735 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.208417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44297 (local address 127.0.98.62:42445)
0619 12:35:55.208573 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:55.208578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:55.208801 (+ 223us) server_negotiation.cc:407] Connection header received
0619 12:35:55.208881 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:55.208884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:55.208939 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:55.209042 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:55.209588 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.210334 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.211507 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.211811 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.211921 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.213156 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:55.213161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:55.213166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:55.213201 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:55.214780 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:55.215343 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:55.215346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:55.215347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:55.215427 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:55.215810 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:55.215813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:55.215814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:55.216094 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:55.216173 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:55.216512 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 12:35:55.216615 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:35:55.216923 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.208332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42445 (local address 127.0.98.2:44297)
0619 12:35:55.208642 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:55.208657 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:55.208793 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:55.209089 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:55.209091 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:55.209328 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:55.209453 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.209461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.210501 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.210505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.211348 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.211361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.211936 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.211939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.212022 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.212629 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:55.212642 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:55.213004 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:55.214906 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:55.214912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:55.214917 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:55.215237 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:55.215586 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:55.215592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:55.215598 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:55.215710 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:55.216203 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:55.216217 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:55.216336 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 12:35:55.216462 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 12:35:55.216651 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":138,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 12:35:55.238814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42445
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:55.353824 8262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:55.354147 8262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:55.354226 8262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:55.358415 8262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:55.358529 8262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:55.358608 8262 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:35:55.358691 8262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:35:55.364202 8262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:55.365330 8262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:55.366405 8262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:35:55.376389 8262 server_base.cc:1034] running on GCE node
W20260619 12:35:55.376354 8267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:55.376333 8270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:55.376322 8268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:55.377082 8262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:55.377800 8262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:55.378995 8262 hybrid_clock.cc:648] HybridClock initialized: now 1781872555378983 us; error 37 us; skew 500 ppm
Jun 19 12:35:55 dist-test-slave-kdrb krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:55.381990 8262 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 12:35:55.383091 8262 webserver.cc:460] Webserver started at http://127.0.98.3:36497/ using document root <none> and password file <none>
I20260619 12:35:55.383750 8262 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:55.383842 8262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:55.384071 8262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:55.386204 8262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "42896eb323d14d81b9137f2dd4954b3f"
format_stamp: "Formatted at 2026-06-19 12:35:55 on dist-test-slave-kdrb"
server_key: "e4d0c60b8e9651fa2bbce456018af6ce"
server_key_iv: "62838bf79d140b9dc60657b602c09bb7"
server_key_version: "encryptionkey@0"
I20260619 12:35:55.386801 8262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "42896eb323d14d81b9137f2dd4954b3f"
format_stamp: "Formatted at 2026-06-19 12:35:55 on dist-test-slave-kdrb"
server_key: "e4d0c60b8e9651fa2bbce456018af6ce"
server_key_iv: "62838bf79d140b9dc60657b602c09bb7"
server_key_version: "encryptionkey@0"
I20260619 12:35:55.390618 8262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:35:55.393102 8277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:55.394294 8262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:35:55.394443 8262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "42896eb323d14d81b9137f2dd4954b3f"
format_stamp: "Formatted at 2026-06-19 12:35:55 on dist-test-slave-kdrb"
server_key: "e4d0c60b8e9651fa2bbce456018af6ce"
server_key_iv: "62838bf79d140b9dc60657b602c09bb7"
server_key_version: "encryptionkey@0"
I20260619 12:35:55.394557 8262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:55.417888 8262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:55.421026 8262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:55.421247 8262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:55.422840 8262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:35:55.422916 8262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:55.422986 8262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:35:55.423038 8262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:35:55 dist-test-slave-kdrb krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:35:55.434722 8262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44967
I20260619 12:35:55.435015 8392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44967 every 8 connection(s)
I20260619 12:35:55.435953 8262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:35:55.437901 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.423694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35629 (local address 127.0.98.62:42445)
0619 12:35:55.423829 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:55.423833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:55.424978 (+ 1145us) server_negotiation.cc:407] Connection header received
0619 12:35:55.426064 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:55.426068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:55.426128 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:55.426191 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:55.427698 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.428296 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.429655 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.429891 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.430011 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.433768 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:55.433772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:55.433775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:55.433809 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:55.435667 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:55.436284 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:55.436290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:55.436295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:55.436406 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:55.436880 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:55.436884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:55.436886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:55.437108 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:55.437185 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:55.437653 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 12:35:55.437759 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:35:55.438824 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.424271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42445 (local address 127.0.98.3:35629)
0619 12:35:55.424802 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:55.424848 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:55.425779 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:55.426619 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:55.426626 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:55.426985 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:55.427537 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.427565 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.428456 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.428460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.429515 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.429525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.430063 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.430067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.430237 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.431002 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:55.431029 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:55.433555 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:55.435792 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:55.435797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:55.435809 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:55.436154 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:55.436532 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:55.436534 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:55.436536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:55.436751 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:55.437194 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:55.437199 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:55.437444 (+ 245us) client_negotiation.cc:769] Sending connection context
0619 12:35:55.437596 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 12:35:55.437888 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":151,"threads_started":1}
I20260619 12:35:55.440114 8393 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42445
I20260619 12:35:55.440280 8393 heartbeater.cc:464] Registering TS with master...
I20260619 12:35:55.440826 8393 heartbeater.cc:511] Master 127.0.98.62:42445 requested a full tablet report, sending...
I20260619 12:35:55.442075 7910 ts_manager.cc:194] Registered new tserver with Master: 42896eb323d14d81b9137f2dd4954b3f (127.0.98.3:44967)
I20260619 12:35:55.442826 7910 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35629
I20260619 12:35:55.445600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 8262
I20260619 12:35:55.445704 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:35:55.446054 392 external_mini_cluster.cc:1428] Setting key cefaec21a4bc7bd00196ce7c2ba0dce4
I20260619 12:35:55.448211 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:35:55.450001 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.442535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40985 (local address 127.0.98.62:42445)
0619 12:35:55.442650 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:55.442654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:55.442899 (+ 245us) server_negotiation.cc:407] Connection header received
0619 12:35:55.443030 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:55.443033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:55.443083 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:55.443170 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:55.443673 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.444190 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.445146 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.445392 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.445503 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.446693 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:55.446697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:55.446699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:55.446724 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:55.448289 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:55.448726 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:55.448731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:55.448733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:55.448846 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:55.449151 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:55.449156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:55.449158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:55.449345 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:55.449451 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:55.449718 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 12:35:55.449828 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:35:55.450008 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.442474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42445 (local address 127.0.98.3:40985)
0619 12:35:55.442803 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:55.442818 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:55.442929 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:55.443187 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:55.443190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:55.443403 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:55.443563 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.443570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.444314 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.444317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.445028 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.445036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.445507 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.445510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.445604 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.446206 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:55.446219 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:55.446557 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:55.448410 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:55.448415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:55.448417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:55.448622 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:55.448944 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:55.448948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:55.448950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:55.449041 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:55.449440 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:55.449444 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:55.449548 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 12:35:55.449654 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 12:35:55.449810 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":110,"threads_started":1}
Jun 19 12:35:55 dist-test-slave-kdrb krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872555, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 12:35:55 dist-test-slave-kdrb krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:35:55 dist-test-slave-kdrb krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 12:35:55.472929 7959 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:35:55.466006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37136 (local address 127.0.98.62:42445)
0619 12:35:55.466130 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:55.466133 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:55.466145 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:35:55.466186 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:55.466189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:55.466229 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:55.466291 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:55.467001 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.467587 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.468386 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.468591 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.468664 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.472724 (+ 4060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37136: BlockingRecv error: recv got EOF from 127.0.0.1:37136 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 12:35:55 dist-test-slave-kdrb krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872555, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:35:55.482836 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.474368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37146 (local address 127.0.98.62:42445)
0619 12:35:55.474518 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:55.474521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:55.474717 (+ 196us) server_negotiation.cc:407] Connection header received
0619 12:35:55.474816 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:55.474819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:55.474861 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:55.474928 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:55.475611 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.476130 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.476951 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.477155 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.477231 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.479816 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:55.479819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:55.479821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:55.479851 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:55.481289 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:55.481709 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:55.481712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:55.481713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:55.481806 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:55.482029 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:55.482031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:55.482032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:55.482207 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:55.482307 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:55.482506 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 12:35:55.482605 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:35:55.486101 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:35:55.489250 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:55.504475 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.498475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55738 (local address 127.0.98.3:44967)
0619 12:35:55.499408 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:55.499422 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:55.499445 (+ 23us) server_negotiation.cc:407] Connection header received
0619 12:35:55.499556 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:55.499561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:55.499763 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:55.499896 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:55.500258 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.501006 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.502647 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.503566 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.503683 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.503879 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:55.504198 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 12:35:55.504306 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":102,"threads_started":1}
I20260619 12:35:55.504460 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.497970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.0.98.1:45309)
0619 12:35:55.498592 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:55.498601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:55.499015 (+ 414us) server_negotiation.cc:407] Connection header received
0619 12:35:55.499104 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:55.499110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:55.499405 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:55.499824 (+ 419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:55.500324 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.501449 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.502914 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.503703 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.503794 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.503916 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:55.504250 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 12:35:55.504346 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":108,"threads_started":1}
I20260619 12:35:55.504493 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.498666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37345 (local address 127.0.0.1:34240)
0619 12:35:55.498963 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:55.498976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:55.499045 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:55.499827 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:55.499843 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:55.499869 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:55.500107 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.500119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.501571 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.501575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.502665 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.502673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.503757 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.503760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.503963 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.504027 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 12:35:55.504136 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 12:35:55.504308 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":116,"thread_start_us":91,"threads_started":1}
I20260619 12:35:55.504732 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.497868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45309 (local address 127.0.0.1:40274)
0619 12:35:55.498320 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:55.498784 (+ 464us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:55.498893 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:55.499983 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:55.499986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:55.500006 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:55.500197 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.500207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.501594 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.501598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.502787 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.502794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.503983 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.503986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.504068 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.504085 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 12:35:55.504179 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 12:35:55.504308 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":76,"thread_start_us":106,"threads_started":1}
I20260619 12:35:55.504998 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.498908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.0.98.2:37345)
0619 12:35:55.499293 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:55.499306 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:55.499328 (+ 22us) server_negotiation.cc:407] Connection header received
0619 12:35:55.499401 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:55.499409 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:55.499638 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:55.500101 (+ 463us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:55.500245 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.501437 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.502852 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.503633 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.504546 (+ 913us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.504720 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:55.504777 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 12:35:55.504895 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":112,"threads_started":1}
I20260619 12:35:55.504493 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.498257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44967 (local address 127.0.0.1:55738)
0619 12:35:55.498710 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:55.498728 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:55.498866 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:55.499968 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:55.499971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:55.499985 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:55.500124 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.500131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.501119 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.501125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.502512 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.502522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.503682 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.503689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.503944 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.504027 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 12:35:55.504142 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 12:35:55.504377 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":274,"spinlock_wait_cycles":2944,"thread_start_us":100,"threads_started":1}
I20260619 12:35:55.508037 8185 tablet_service.cc:1467] Processing CreateTablet for tablet 4865c0f3b5f74536831bb7fd373dc3ef (DEFAULT_TABLE table=test-table [id=41a192b7945941dcb2f077a9d2ca1490]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:55.508055 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 4865c0f3b5f74536831bb7fd373dc3ef (DEFAULT_TABLE table=test-table [id=41a192b7945941dcb2f077a9d2ca1490]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:55.508054 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 4865c0f3b5f74536831bb7fd373dc3ef (DEFAULT_TABLE table=test-table [id=41a192b7945941dcb2f077a9d2ca1490]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:55.509040 8185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4865c0f3b5f74536831bb7fd373dc3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:55.509099 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4865c0f3b5f74536831bb7fd373dc3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:55.509094 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4865c0f3b5f74536831bb7fd373dc3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:55.516096 8421 tablet_bootstrap.cc:492] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e: Bootstrap starting.
I20260619 12:35:55.516770 8422 tablet_bootstrap.cc:492] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4: Bootstrap starting.
I20260619 12:35:55.517450 8423 tablet_bootstrap.cc:492] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f: Bootstrap starting.
I20260619 12:35:55.518858 8421 tablet_bootstrap.cc:654] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:55.518858 8422 tablet_bootstrap.cc:654] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:55.519205 8423 tablet_bootstrap.cc:654] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:55.519835 8422 log.cc:824] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:55.519845 8421 log.cc:824] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:55.520043 8423 log.cc:824] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f: Log is configured to *not* fsync() on all Append() calls
I20260619 12:35:55.521909 8423 tablet_bootstrap.cc:492] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f: No bootstrap required, opened a new log
I20260619 12:35:55.521919 8421 tablet_bootstrap.cc:492] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e: No bootstrap required, opened a new log
I20260619 12:35:55.522121 8423 ts_tablet_manager.cc:1397] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260619 12:35:55.522123 8421 ts_tablet_manager.cc:1397] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:35:55.523274 8422 tablet_bootstrap.cc:492] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4: No bootstrap required, opened a new log
I20260619 12:35:55.523504 8422 ts_tablet_manager.cc:1397] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 12:35:55.525276 8423 raft_consensus.cc:348] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.525552 8423 raft_consensus.cc:374] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:55.525604 8423 raft_consensus.cc:729] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42896eb323d14d81b9137f2dd4954b3f, State: Initialized, Role: FOLLOWER
I20260619 12:35:55.526062 8423 consensus_queue.cc:260] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.526088 8421 raft_consensus.cc:348] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.526361 8421 raft_consensus.cc:374] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:55.526419 8421 raft_consensus.cc:729] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d57187001f6464b9a5721b92d3b102e, State: Initialized, Role: FOLLOWER
I20260619 12:35:55.526827 8393 heartbeater.cc:503] Master 127.0.98.62:42445 was elected leader, sending a full tablet report...
I20260619 12:35:55.526858 8421 consensus_queue.cc:260] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.527163 8423 ts_tablet_manager.cc:1428] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:35:55.527843 8421 ts_tablet_manager.cc:1428] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 12:35:55.528292 8253 heartbeater.cc:503] Master 127.0.98.62:42445 was elected leader, sending a full tablet report...
I20260619 12:35:55.527844 8422 raft_consensus.cc:348] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.528461 8422 raft_consensus.cc:374] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:55.528517 8422 raft_consensus.cc:729] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 680b9950669e4d98abdc4b6ccf6fe0b4, State: Initialized, Role: FOLLOWER
I20260619 12:35:55.528929 8422 consensus_queue.cc:260] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.529552 8113 heartbeater.cc:503] Master 127.0.98.62:42445 was elected leader, sending a full tablet report...
I20260619 12:35:55.530171 8422 ts_tablet_manager.cc:1428] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260619 12:35:55.686978 8394 tablet.cc:2369] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:35:55.701846 8254 tablet.cc:2369] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:35:55.726547 8114 tablet.cc:2369] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:35:55.742482 8428 raft_consensus.cc:482] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:55.742725 8428 raft_consensus.cc:504] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.744036 8428 leader_election.cc:288] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 680b9950669e4d98abdc4b6ccf6fe0b4 (127.0.98.1:45309), 42896eb323d14d81b9137f2dd4954b3f (127.0.98.3:44967)
I20260619 12:35:55.748594 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.744531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42165 (local address 127.0.98.3:44967)
0619 12:35:55.744751 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:55.744758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:55.744940 (+ 182us) server_negotiation.cc:407] Connection header received
0619 12:35:55.745090 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:55.745092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:55.745175 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:55.745245 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:55.745663 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.746221 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.747537 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.747935 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.748023 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.748103 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:55.748350 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 12:35:55.748455 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260619 12:35:55.748665 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.744549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45449 (local address 127.0.98.1:45309)
0619 12:35:55.744693 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:55.744698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:55.744941 (+ 243us) server_negotiation.cc:407] Connection header received
0619 12:35:55.745088 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:55.745092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:55.745154 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:55.745243 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:55.745754 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.746379 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.747509 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.748024 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.748139 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.748220 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:55.748443 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 12:35:55.748545 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:35:55.748631 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.744383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44967 (local address 127.0.98.2:42165)
0619 12:35:55.744739 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:55.744758 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:55.744933 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:55.745371 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:55.745374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:55.745403 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:55.745537 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.745543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.746353 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.746357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.747350 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.747360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.748078 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.748086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.748198 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.748224 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 12:35:55.748307 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 12:35:55.748432 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":30,"spinlock_wait_cycles":896,"thread_start_us":81,"threads_started":1}
I20260619 12:35:55.749004 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.744383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45309 (local address 127.0.98.2:45449)
0619 12:35:55.744818 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:55.744834 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:55.744969 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:55.745349 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:55.745354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:55.745385 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:55.745578 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.745584 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.746488 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.746491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.747326 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.747335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.748160 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.748163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.748239 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.748256 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 12:35:55.748786 (+ 530us) client_negotiation.cc:240] Negotiation successful
0619 12:35:55.748876 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":88,"spinlock_wait_cycles":22784,"thread_start_us":80,"threads_started":1}
I20260619 12:35:55.749452 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4865c0f3b5f74536831bb7fd373dc3ef" candidate_uuid: "9d57187001f6464b9a5721b92d3b102e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42896eb323d14d81b9137f2dd4954b3f" is_pre_election: true
I20260619 12:35:55.749720 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4865c0f3b5f74536831bb7fd373dc3ef" candidate_uuid: "9d57187001f6464b9a5721b92d3b102e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" is_pre_election: true
I20260619 12:35:55.749876 8347 raft_consensus.cc:2454] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d57187001f6464b9a5721b92d3b102e in term 0.
I20260619 12:35:55.750032 8067 raft_consensus.cc:2454] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d57187001f6464b9a5721b92d3b102e in term 0.
I20260619 12:35:55.750505 8139 leader_election.cc:302] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 42896eb323d14d81b9137f2dd4954b3f, 680b9950669e4d98abdc4b6ccf6fe0b4, 9d57187001f6464b9a5721b92d3b102e; no voters:
I20260619 12:35:55.750823 8428 raft_consensus.cc:2781] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:55.750914 8428 raft_consensus.cc:482] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:55.750960 8428 raft_consensus.cc:3037] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:55.752132 8428 raft_consensus.cc:504] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.752532 8428 leader_election.cc:288] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [CANDIDATE]: Term 1 election: Requested vote from peers 680b9950669e4d98abdc4b6ccf6fe0b4 (127.0.98.1:45309), 42896eb323d14d81b9137f2dd4954b3f (127.0.98.3:44967)
I20260619 12:35:55.752962 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4865c0f3b5f74536831bb7fd373dc3ef" candidate_uuid: "9d57187001f6464b9a5721b92d3b102e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42896eb323d14d81b9137f2dd4954b3f"
I20260619 12:35:55.752969 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4865c0f3b5f74536831bb7fd373dc3ef" candidate_uuid: "9d57187001f6464b9a5721b92d3b102e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4"
I20260619 12:35:55.753085 8347 raft_consensus.cc:3037] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:55.753118 8067 raft_consensus.cc:3037] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:55.754204 8067 raft_consensus.cc:2454] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d57187001f6464b9a5721b92d3b102e in term 1.
I20260619 12:35:55.754595 8140 leader_election.cc:302] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 680b9950669e4d98abdc4b6ccf6fe0b4, 9d57187001f6464b9a5721b92d3b102e; no voters:
I20260619 12:35:55.754737 8347 raft_consensus.cc:2454] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d57187001f6464b9a5721b92d3b102e in term 1.
I20260619 12:35:55.754796 8428 raft_consensus.cc:2781] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:55.755041 8428 raft_consensus.cc:686] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [term 1 LEADER]: Becoming Leader. State: Replica: 9d57187001f6464b9a5721b92d3b102e, State: Running, Role: LEADER
I20260619 12:35:55.755420 8428 consensus_queue.cc:237] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.758836 7909 catalog_manager.cc:5495] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d57187001f6464b9a5721b92d3b102e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9d57187001f6464b9a5721b92d3b102e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20260619 12:35:55.798215 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.794331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.0.98.2:37345)
0619 12:35:55.794474 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:55.794478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:55.794492 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:35:55.794598 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:55.794600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:55.794659 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:55.794752 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:55.794985 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.795523 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.796497 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.796740 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.796837 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.797158 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:55.797680 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:55.797872 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:55.797905 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 12:35:55.797969 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:35:55.804641 8347 raft_consensus.cc:1261] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f [term 1 FOLLOWER]: Refusing update from remote peer 9d57187001f6464b9a5721b92d3b102e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:55.804641 8067 raft_consensus.cc:1261] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Refusing update from remote peer 9d57187001f6464b9a5721b92d3b102e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:55.805433 8432 consensus_queue.cc:1035] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [LEADER]: Connected to new peer: Peer: permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:55.805630 8428 consensus_queue.cc:1035] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [LEADER]: Connected to new peer: Peer: permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:55.813438 8436 mvcc.cc:205] Tried to move back new op lower bound from 7298549988568043520 to 7298549988377391104. Current Snapshot: MvccSnapshot[applied={T|T < 7298549988568043520}]
I20260619 12:35:55.817457 8437 mvcc.cc:205] Tried to move back new op lower bound from 7298549988568043520 to 7298549988377391104. Current Snapshot: MvccSnapshot[applied={T|T < 7298549988568043520 or (T in {7298549988568043520})}]
I20260619 12:35:55.821969 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:55.822197 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:55.824671 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359: Sending DeleteTablet for 3 replicas of tablet 4865c0f3b5f74536831bb7fd373dc3ef
I20260619 12:35:55.825531 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet 4865c0f3b5f74536831bb7fd373dc3ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55738
I20260619 12:35:55.825563 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet 4865c0f3b5f74536831bb7fd373dc3ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40274
I20260619 12:35:55.826082 8446 tablet_replica.cc:330] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f: stopping tablet replica
I20260619 12:35:55.826172 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 12:35:55.826239 8185 tablet_service.cc:1514] Processing DeleteTablet for tablet 4865c0f3b5f74536831bb7fd373dc3ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34240
I20260619 12:35:55.826423 8446 raft_consensus.cc:2229] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 12:35:55.826588 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:55.826776 8446 raft_consensus.cc:2258] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:55.827085 8445 tablet_replica.cc:330] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4: stopping tablet replica
I20260619 12:35:55.827396 8445 raft_consensus.cc:2229] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:55.827646 8447 tablet_replica.cc:330] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e: stopping tablet replica
I20260619 12:35:55.827906 8447 raft_consensus.cc:2229] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:35:55.829047 8446 ts_tablet_manager.cc:1905] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:55.829113 8445 raft_consensus.cc:2258] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:55.829442 8447 raft_consensus.cc:2258] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:55.830790 8447 ts_tablet_manager.cc:1905] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:55.832297 8446 ts_tablet_manager.cc:1918] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:55.832489 8446 log.cc:1196] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/4865c0f3b5f74536831bb7fd373dc3ef
I20260619 12:35:55.832901 8446 ts_tablet_manager.cc:1939] T 4865c0f3b5f74536831bb7fd373dc3ef P 42896eb323d14d81b9137f2dd4954b3f: Deleting consensus metadata
I20260619 12:35:55.833058 8047 tablet_service.cc:1467] Processing CreateTablet for tablet a6969e5953cc4a5f964df7d2b654eb95 (DEFAULT_TABLE table=test-table [id=92b9f26247bd4c789a117d60064da639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:55.833350 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6969e5953cc4a5f964df7d2b654eb95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:55.833705 8447 ts_tablet_manager.cc:1918] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:55.833807 8447 log.cc:1196] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/4865c0f3b5f74536831bb7fd373dc3ef
I20260619 12:35:55.834112 8447 ts_tablet_manager.cc:1939] T 4865c0f3b5f74536831bb7fd373dc3ef P 9d57187001f6464b9a5721b92d3b102e: Deleting consensus metadata
I20260619 12:35:55.834707 7895 catalog_manager.cc:4841] TS 42896eb323d14d81b9137f2dd4954b3f (127.0.98.3:44967): tablet 4865c0f3b5f74536831bb7fd373dc3ef (table test-table [id=41a192b7945941dcb2f077a9d2ca1490]) successfully deleted
I20260619 12:35:55.835186 7896 catalog_manager.cc:4841] TS 9d57187001f6464b9a5721b92d3b102e (127.0.98.2:37345): tablet 4865c0f3b5f74536831bb7fd373dc3ef (table test-table [id=41a192b7945941dcb2f077a9d2ca1490]) successfully deleted
I20260619 12:35:55.835783 8422 tablet_bootstrap.cc:492] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4: Bootstrap starting.
I20260619 12:35:55.835804 8327 tablet_service.cc:1467] Processing CreateTablet for tablet a6969e5953cc4a5f964df7d2b654eb95 (DEFAULT_TABLE table=test-table [id=92b9f26247bd4c789a117d60064da639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:55.836040 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6969e5953cc4a5f964df7d2b654eb95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:55.836813 8422 tablet_bootstrap.cc:654] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:55.838177 8422 tablet_bootstrap.cc:492] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4: No bootstrap required, opened a new log
I20260619 12:35:55.838260 8422 ts_tablet_manager.cc:1397] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:35:55.838423 8423 tablet_bootstrap.cc:492] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f: Bootstrap starting.
I20260619 12:35:55.838660 8185 tablet_service.cc:1467] Processing CreateTablet for tablet a6969e5953cc4a5f964df7d2b654eb95 (DEFAULT_TABLE table=test-table [id=92b9f26247bd4c789a117d60064da639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:55.838747 8422 raft_consensus.cc:348] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.838855 8422 raft_consensus.cc:374] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:55.838889 8422 raft_consensus.cc:729] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 680b9950669e4d98abdc4b6ccf6fe0b4, State: Initialized, Role: FOLLOWER
I20260619 12:35:55.838979 8185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6969e5953cc4a5f964df7d2b654eb95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:55.839062 8422 consensus_queue.cc:260] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.839434 8423 tablet_bootstrap.cc:654] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:55.840636 8423 tablet_bootstrap.cc:492] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f: No bootstrap required, opened a new log
I20260619 12:35:55.840709 8423 ts_tablet_manager.cc:1397] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:35:55.841202 8423 raft_consensus.cc:348] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.841324 8423 raft_consensus.cc:374] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:55.841356 8423 raft_consensus.cc:729] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42896eb323d14d81b9137f2dd4954b3f, State: Initialized, Role: FOLLOWER
I20260619 12:35:55.841471 8423 consensus_queue.cc:260] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.841523 8422 ts_tablet_manager.cc:1428] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260619 12:35:55.841717 8423 ts_tablet_manager.cc:1428] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:35:55.841967 8421 tablet_bootstrap.cc:492] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e: Bootstrap starting.
I20260619 12:35:55.842940 8421 tablet_bootstrap.cc:654] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:55.843870 8445 ts_tablet_manager.cc:1905] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:55.844205 8421 tablet_bootstrap.cc:492] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e: No bootstrap required, opened a new log
I20260619 12:35:55.844316 8421 ts_tablet_manager.cc:1397] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:35:55.844818 8421 raft_consensus.cc:348] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.844954 8421 raft_consensus.cc:374] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:55.844993 8421 raft_consensus.cc:729] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d57187001f6464b9a5721b92d3b102e, State: Initialized, Role: FOLLOWER
I20260619 12:35:55.845139 8421 consensus_queue.cc:260] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.845474 8421 ts_tablet_manager.cc:1428] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:35:55.847630 8445 ts_tablet_manager.cc:1918] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:55.847810 8445 log.cc:1196] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/wal/wals/4865c0f3b5f74536831bb7fd373dc3ef
I20260619 12:35:55.848160 8445 ts_tablet_manager.cc:1939] T 4865c0f3b5f74536831bb7fd373dc3ef P 680b9950669e4d98abdc4b6ccf6fe0b4: Deleting consensus metadata
I20260619 12:35:55.849493 7896 catalog_manager.cc:4841] TS 680b9950669e4d98abdc4b6ccf6fe0b4 (127.0.98.1:45309): tablet 4865c0f3b5f74536831bb7fd373dc3ef (table test-table [id=41a192b7945941dcb2f077a9d2ca1490]) successfully deleted
I20260619 12:35:55.851822 8429 raft_consensus.cc:482] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:55.851958 8429 raft_consensus.cc:504] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.853286 8429 leader_election.cc:288] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d57187001f6464b9a5721b92d3b102e (127.0.98.2:37345), 42896eb323d14d81b9137f2dd4954b3f (127.0.98.3:44967)
I20260619 12:35:55.857625 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.853610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50241 (local address 127.0.98.3:44967)
0619 12:35:55.853736 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:55.853743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:55.854077 (+ 334us) server_negotiation.cc:407] Connection header received
0619 12:35:55.854222 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:55.854225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:55.854275 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:55.854339 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:55.854689 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.855185 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.856634 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.857050 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.857172 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.857243 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:55.857420 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 12:35:55.857495 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:35:55.857636 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.853521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44967 (local address 127.0.98.1:50241)
0619 12:35:55.854004 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:55.854015 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:55.854135 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:55.854437 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:55.854439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:55.854453 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:55.854574 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.854579 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.855297 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.855302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.856498 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.856510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.857176 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.857179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.857284 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.857307 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:35:55.857405 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 12:35:55.857483 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":109,"threads_started":1}
I20260619 12:35:55.858160 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6969e5953cc4a5f964df7d2b654eb95" candidate_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42896eb323d14d81b9137f2dd4954b3f" is_pre_election: true
I20260619 12:35:55.858333 8347 raft_consensus.cc:2454] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 680b9950669e4d98abdc4b6ccf6fe0b4 in term 0.
I20260619 12:35:55.858718 7999 leader_election.cc:302] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42896eb323d14d81b9137f2dd4954b3f, 680b9950669e4d98abdc4b6ccf6fe0b4; no voters:
I20260619 12:35:55.858953 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.853826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37043 (local address 127.0.98.2:37345)
0619 12:35:55.853968 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:55.853974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:55.853991 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:35:55.854140 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:55.854143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:55.854191 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:55.854270 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:55.854548 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.855315 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.856878 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.857490 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.857575 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.857635 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:55.858759 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0619 12:35:55.858840 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 12:35:55.858961 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.853459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37345 (local address 127.0.98.1:37043)
0619 12:35:55.853901 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:55.853916 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:55.854023 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:55.854276 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:55.854279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:55.854304 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:55.854437 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.854444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.855452 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.855457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.856714 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.856725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.858501 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.858507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.858613 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.858636 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:35:55.858719 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 12:35:55.858789 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":96,"threads_started":1}
I20260619 12:35:55.859033 8429 raft_consensus.cc:2781] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:55.859086 8429 raft_consensus.cc:482] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:55.859128 8429 raft_consensus.cc:3037] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:55.859515 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6969e5953cc4a5f964df7d2b654eb95" candidate_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d57187001f6464b9a5721b92d3b102e" is_pre_election: true
I20260619 12:35:55.859745 8207 raft_consensus.cc:2454] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 680b9950669e4d98abdc4b6ccf6fe0b4 in term 0.
I20260619 12:35:55.860448 8429 raft_consensus.cc:504] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.860860 8429 leader_election.cc:288] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [CANDIDATE]: Term 1 election: Requested vote from peers 9d57187001f6464b9a5721b92d3b102e (127.0.98.2:37345), 42896eb323d14d81b9137f2dd4954b3f (127.0.98.3:44967)
I20260619 12:35:55.861219 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6969e5953cc4a5f964df7d2b654eb95" candidate_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42896eb323d14d81b9137f2dd4954b3f"
I20260619 12:35:55.861331 8347 raft_consensus.cc:3037] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:55.861939 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6969e5953cc4a5f964df7d2b654eb95" candidate_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d57187001f6464b9a5721b92d3b102e"
I20260619 12:35:55.862040 8207 raft_consensus.cc:3037] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:55.862696 8347 raft_consensus.cc:2454] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 680b9950669e4d98abdc4b6ccf6fe0b4 in term 1.
I20260619 12:35:55.862919 8207 raft_consensus.cc:2454] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 680b9950669e4d98abdc4b6ccf6fe0b4 in term 1.
I20260619 12:35:55.863183 8000 leader_election.cc:302] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 680b9950669e4d98abdc4b6ccf6fe0b4, 9d57187001f6464b9a5721b92d3b102e; no voters:
I20260619 12:35:55.863353 8429 raft_consensus.cc:2781] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:55.863582 8429 raft_consensus.cc:686] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 LEADER]: Becoming Leader. State: Replica: 680b9950669e4d98abdc4b6ccf6fe0b4, State: Running, Role: LEADER
I20260619 12:35:55.863858 8429 consensus_queue.cc:237] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.866182 7910 catalog_manager.cc:5495] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 680b9950669e4d98abdc4b6ccf6fe0b4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } health_report { overall_health: UNKNOWN } } }
Jun 19 12:35:55 dist-test-slave-kdrb krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872554, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:35:55.911079 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.901417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.0.98.62:42445)
0619 12:35:55.901561 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:55.901566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:55.901821 (+ 255us) server_negotiation.cc:407] Connection header received
0619 12:35:55.901988 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:55.901991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:55.902039 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:55.902125 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:55.902948 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.903580 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.904435 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.904669 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.904796 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.907427 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:55.907431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:55.907433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:55.907462 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:55.909055 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:55.909638 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:55.909642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:55.909644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:55.909736 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:55.910041 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:55.910048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:55.910050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:55.910259 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:55.910376 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:55.910701 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 12:35:55.910839 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":53}
I20260619 12:35:55.911118 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:55.901363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42445 (local address 127.0.0.1:37160)
0619 12:35:55.901706 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:55.901725 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:55.901884 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:55.902170 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:55.902174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:55.902666 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:35:55.902786 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.902794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.903706 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.903712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.904294 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:55.904303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:55.904804 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:55.904808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:55.904918 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:55.905623 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:35:55.905636 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:35:55.907278 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:35:55.909226 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:55.909239 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:55.909247 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:55.909514 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:55.909839 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:35:55.909845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:35:55.909849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:35:55.909948 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:35:55.910376 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:35:55.910385 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:35:55.910512 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 12:35:55.910662 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 12:35:55.910919 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":102,"thread_start_us":157,"threads_started":1}
I20260619 12:35:55.915292 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37160:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 12:35:55.916103 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:55.921294 8185 tablet_service.cc:1467] Processing CreateTablet for tablet b846f52889f54cacad0b3b6337625223 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=639bc30eb76f447a91a92ce573875d02]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:35:55.921294 8327 tablet_service.cc:1467] Processing CreateTablet for tablet b846f52889f54cacad0b3b6337625223 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=639bc30eb76f447a91a92ce573875d02]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:35:55.921345 8047 tablet_service.cc:1467] Processing CreateTablet for tablet b846f52889f54cacad0b3b6337625223 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=639bc30eb76f447a91a92ce573875d02]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:35:55.921708 8185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b846f52889f54cacad0b3b6337625223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:55.921712 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b846f52889f54cacad0b3b6337625223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:55.921878 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b846f52889f54cacad0b3b6337625223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:55.924643 8421 tablet_bootstrap.cc:492] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e: Bootstrap starting.
I20260619 12:35:55.925443 8421 tablet_bootstrap.cc:654] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:55.925634 8422 tablet_bootstrap.cc:492] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4: Bootstrap starting.
I20260619 12:35:55.926705 8422 tablet_bootstrap.cc:654] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:55.926947 8421 tablet_bootstrap.cc:492] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e: No bootstrap required, opened a new log
I20260619 12:35:55.927040 8421 ts_tablet_manager.cc:1397] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 12:35:55.927595 8421 raft_consensus.cc:348] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.927749 8421 raft_consensus.cc:374] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:55.927790 8421 raft_consensus.cc:729] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d57187001f6464b9a5721b92d3b102e, State: Initialized, Role: FOLLOWER
I20260619 12:35:55.927919 8421 consensus_queue.cc:260] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.928031 8422 tablet_bootstrap.cc:492] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4: No bootstrap required, opened a new log
I20260619 12:35:55.928110 8422 ts_tablet_manager.cc:1397] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:35:55.928552 8442 tablet_replica.cc:439] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } } }
I20260619 12:35:55.928568 8422 raft_consensus.cc:348] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.928674 8422 raft_consensus.cc:374] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:55.928709 8442 tablet_replica.cc:442] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:55.928704 8422 raft_consensus.cc:729] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 680b9950669e4d98abdc4b6ccf6fe0b4, State: Initialized, Role: FOLLOWER
I20260619 12:35:55.928894 8422 consensus_queue.cc:260] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.929121 8429 tablet_replica.cc:439] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } } }
I20260619 12:35:55.929255 8429 tablet_replica.cc:442] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:55.929504 8422 ts_tablet_manager.cc:1428] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:35:55.930531 8421 ts_tablet_manager.cc:1428] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:35:55.931109 8423 tablet_bootstrap.cc:492] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f: Bootstrap starting.
I20260619 12:35:55.932312 8423 tablet_bootstrap.cc:654] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:55.933770 8423 tablet_bootstrap.cc:492] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f: No bootstrap required, opened a new log
I20260619 12:35:55.933868 8423 ts_tablet_manager.cc:1397] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:35:55.934315 8423 raft_consensus.cc:348] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.934458 8423 raft_consensus.cc:374] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:55.934505 8423 raft_consensus.cc:729] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42896eb323d14d81b9137f2dd4954b3f, State: Initialized, Role: FOLLOWER
I20260619 12:35:55.934640 8423 consensus_queue.cc:260] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:55.934953 8423 ts_tablet_manager.cc:1428] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 12:35:55.934907 8427 tablet_replica.cc:439] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } } }
I20260619 12:35:55.935063 8427 tablet_replica.cc:442] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:56.232024 8429 raft_consensus.cc:482] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:56.232174 8429 raft_consensus.cc:504] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.232576 8429 leader_election.cc:288] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d57187001f6464b9a5721b92d3b102e (127.0.98.2:37345), 42896eb323d14d81b9137f2dd4954b3f (127.0.98.3:44967)
I20260619 12:35:56.233168 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b846f52889f54cacad0b3b6337625223" candidate_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42896eb323d14d81b9137f2dd4954b3f" is_pre_election: true
I20260619 12:35:56.233193 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b846f52889f54cacad0b3b6337625223" candidate_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d57187001f6464b9a5721b92d3b102e" is_pre_election: true
I20260619 12:35:56.233343 8347 raft_consensus.cc:2454] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 680b9950669e4d98abdc4b6ccf6fe0b4 in term 0.
I20260619 12:35:56.233350 8207 raft_consensus.cc:2454] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 680b9950669e4d98abdc4b6ccf6fe0b4 in term 0.
I20260619 12:35:56.233702 7999 leader_election.cc:302] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42896eb323d14d81b9137f2dd4954b3f, 680b9950669e4d98abdc4b6ccf6fe0b4; no voters:
I20260619 12:35:56.233925 8429 raft_consensus.cc:2781] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:56.233989 8429 raft_consensus.cc:482] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:56.234015 8429 raft_consensus.cc:3037] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:56.234979 8429 raft_consensus.cc:504] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.235365 8429 leader_election.cc:288] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [CANDIDATE]: Term 1 election: Requested vote from peers 9d57187001f6464b9a5721b92d3b102e (127.0.98.2:37345), 42896eb323d14d81b9137f2dd4954b3f (127.0.98.3:44967)
I20260619 12:35:56.235770 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b846f52889f54cacad0b3b6337625223" candidate_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42896eb323d14d81b9137f2dd4954b3f"
I20260619 12:35:56.235774 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b846f52889f54cacad0b3b6337625223" candidate_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d57187001f6464b9a5721b92d3b102e"
I20260619 12:35:56.235879 8347 raft_consensus.cc:3037] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:56.235913 8207 raft_consensus.cc:3037] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:56.236809 8347 raft_consensus.cc:2454] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 680b9950669e4d98abdc4b6ccf6fe0b4 in term 1.
I20260619 12:35:56.236873 8207 raft_consensus.cc:2454] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 680b9950669e4d98abdc4b6ccf6fe0b4 in term 1.
I20260619 12:35:56.237207 7999 leader_election.cc:302] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42896eb323d14d81b9137f2dd4954b3f, 680b9950669e4d98abdc4b6ccf6fe0b4; no voters:
I20260619 12:35:56.237414 8429 raft_consensus.cc:2781] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:56.237524 8429 raft_consensus.cc:686] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 LEADER]: Becoming Leader. State: Replica: 680b9950669e4d98abdc4b6ccf6fe0b4, State: Running, Role: LEADER
I20260619 12:35:56.237668 8429 consensus_queue.cc:237] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.238337 8453 tablet_replica.cc:439] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4: TxnStatusTablet state changed. Reason: New leader 680b9950669e4d98abdc4b6ccf6fe0b4. Latest consensus state: current_term: 1 leader_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } } }
I20260619 12:35:56.238462 8453 tablet_replica.cc:442] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:35:56.238749 8465 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 12:35:56.238921 8465 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 12:35:56.239602 7910 catalog_manager.cc:5495] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 680b9950669e4d98abdc4b6ccf6fe0b4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20260619 12:35:56.267433 8453 consensus_queue.cc:1035] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:56.271847 8453 consensus_queue.cc:1035] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:56.301267 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:56.297085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45309 (local address 127.0.0.1:40278)
0619 12:35:56.297318 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:56.297336 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:56.297471 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:56.297905 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:56.297908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:56.297917 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:35:56.298039 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:56.298045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.298847 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.298853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:56.299555 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:56.299565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.300045 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.300048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:56.300147 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:56.300200 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:35:56.300938 (+ 738us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:35:56.300956 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 12:35:56.301027 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 12:35:56.301121 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 12:35:56.301278 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:56.297212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.0.98.1:45309)
0619 12:35:56.297485 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:56.297489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:56.297505 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:35:56.297678 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:56.297682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:56.297753 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:56.297905 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:56.298156 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.298720 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.299684 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.299920 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.300041 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:56.300367 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:56.300838 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:56.300968 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:56.301036 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 12:35:56.301092 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260619 12:35:56.304570 8347 raft_consensus.cc:1261] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f [term 1 FOLLOWER]: Refusing update from remote peer 680b9950669e4d98abdc4b6ccf6fe0b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:56.304575 8207 raft_consensus.cc:1261] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e [term 1 FOLLOWER]: Refusing update from remote peer 680b9950669e4d98abdc4b6ccf6fe0b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:56.305073 8453 consensus_queue.cc:1035] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:56.305341 8429 consensus_queue.cc:1035] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:56.306768 8427 tablet_replica.cc:439] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f: TxnStatusTablet state changed. Reason: New leader 680b9950669e4d98abdc4b6ccf6fe0b4. Latest consensus state: current_term: 1 leader_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } } }
I20260619 12:35:56.306769 8442 tablet_replica.cc:439] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e: TxnStatusTablet state changed. Reason: New leader 680b9950669e4d98abdc4b6ccf6fe0b4. Latest consensus state: current_term: 1 leader_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } } }
I20260619 12:35:56.306892 8442 tablet_replica.cc:442] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:56.306893 8427 tablet_replica.cc:442] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:56.309460 8429 tablet_replica.cc:439] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } } }
I20260619 12:35:56.309589 8429 tablet_replica.cc:442] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:35:56.309688 8429 tablet_replica.cc:439] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } } }
I20260619 12:35:56.309746 8429 tablet_replica.cc:442] T b846f52889f54cacad0b3b6337625223 P 680b9950669e4d98abdc4b6ccf6fe0b4: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:35:56.309798 8427 tablet_replica.cc:439] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } } }
I20260619 12:35:56.309912 8427 tablet_replica.cc:442] T b846f52889f54cacad0b3b6337625223 P 42896eb323d14d81b9137f2dd4954b3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:56.311357 8435 mvcc.cc:205] Tried to move back new op lower bound from 7298549990619070464 to 7298549990350843904. Current Snapshot: MvccSnapshot[applied={T|T < 7298549990619070464 or (T in {7298549990619070464})}]
I20260619 12:35:56.311599 8442 tablet_replica.cc:439] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } } }
I20260619 12:35:56.311712 8442 tablet_replica.cc:442] T b846f52889f54cacad0b3b6337625223 P 9d57187001f6464b9a5721b92d3b102e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:35:56.318363 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:56.314814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40294 (local address 127.0.98.1:45309)
0619 12:35:56.314952 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:56.314957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:56.315092 (+ 135us) server_negotiation.cc:407] Connection header received
0619 12:35:56.315260 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:56.315263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:56.315331 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:56.315414 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:56.315710 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.316220 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.317307 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.317532 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.317642 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:56.317898 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:56.317999 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:56.318079 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:56.318167 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 12:35:56.318220 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 12:35:56.330350 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:56.326574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55795 (local address 127.0.98.1:45309)
0619 12:35:56.326723 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:56.326728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:56.326975 (+ 247us) server_negotiation.cc:407] Connection header received
0619 12:35:56.327113 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:56.327115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:56.327158 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:56.327215 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:56.327515 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.328012 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.329242 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.329489 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.329573 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:56.329897 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:56.329970 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:56.330028 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:56.330147 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 12:35:56.330214 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":40}
I20260619 12:35:56.330423 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:56.326500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45309 (local address 127.0.98.1:55795)
0619 12:35:56.326888 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:56.326903 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:56.327011 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:56.327251 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:56.327254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:56.327262 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:35:56.327374 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:56.327380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.328125 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.328129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:56.329106 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:56.329117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.329604 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.329608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:56.329692 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:56.329750 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:35:56.330053 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:35:56.330059 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:35:56.330126 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 12:35:56.330246 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":20,"thread_start_us":104,"threads_started":1}
I20260619 12:35:56.379930 7908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:56.380149 7908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:56.382339 7908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359: Sending DeleteTablet for 3 replicas of tablet a6969e5953cc4a5f964df7d2b654eb95
I20260619 12:35:56.383061 8185 tablet_service.cc:1514] Processing DeleteTablet for tablet a6969e5953cc4a5f964df7d2b654eb95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34240
I20260619 12:35:56.383265 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet a6969e5953cc4a5f964df7d2b654eb95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55738
I20260619 12:35:56.383387 8480 tablet_replica.cc:330] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e: stopping tablet replica
I20260619 12:35:56.383576 8480 raft_consensus.cc:2229] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:56.383579 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet a6969e5953cc4a5f964df7d2b654eb95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40274
I20260619 12:35:56.383620 8481 tablet_replica.cc:330] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f: stopping tablet replica
I20260619 12:35:56.383711 8480 raft_consensus.cc:2258] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:56.383739 8481 raft_consensus.cc:2229] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:56.383862 8481 raft_consensus.cc:2258] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:56.384572 8481 ts_tablet_manager.cc:1905] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:56.387378 8480 ts_tablet_manager.cc:1905] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:56.387626 8482 tablet_replica.cc:330] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4: stopping tablet replica
I20260619 12:35:56.387761 8482 raft_consensus.cc:2229] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:35:56.387763 8481 ts_tablet_manager.cc:1918] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:35:56.387840 8481 log.cc:1196] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/a6969e5953cc4a5f964df7d2b654eb95
I20260619 12:35:56.388085 8482 raft_consensus.cc:2258] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:56.388139 8481 ts_tablet_manager.cc:1939] T a6969e5953cc4a5f964df7d2b654eb95 P 42896eb323d14d81b9137f2dd4954b3f: Deleting consensus metadata
I20260619 12:35:56.388753 8482 ts_tablet_manager.cc:1905] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:56.388926 7895 catalog_manager.cc:4841] TS 42896eb323d14d81b9137f2dd4954b3f (127.0.98.3:44967): tablet a6969e5953cc4a5f964df7d2b654eb95 (table test-table [id=92b9f26247bd4c789a117d60064da639]) successfully deleted
I20260619 12:35:56.390977 8480 ts_tablet_manager.cc:1918] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:35:56.391069 8480 log.cc:1196] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/a6969e5953cc4a5f964df7d2b654eb95
I20260619 12:35:56.391326 8480 ts_tablet_manager.cc:1939] T a6969e5953cc4a5f964df7d2b654eb95 P 9d57187001f6464b9a5721b92d3b102e: Deleting consensus metadata
I20260619 12:35:56.391739 8482 ts_tablet_manager.cc:1918] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:35:56.391839 8482 log.cc:1196] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/wal/wals/a6969e5953cc4a5f964df7d2b654eb95
I20260619 12:35:56.392117 8482 ts_tablet_manager.cc:1939] T a6969e5953cc4a5f964df7d2b654eb95 P 680b9950669e4d98abdc4b6ccf6fe0b4: Deleting consensus metadata
I20260619 12:35:56.392364 7896 catalog_manager.cc:4841] TS 9d57187001f6464b9a5721b92d3b102e (127.0.98.2:37345): tablet a6969e5953cc4a5f964df7d2b654eb95 (table test-table [id=92b9f26247bd4c789a117d60064da639]) successfully deleted
I20260619 12:35:56.392860 7896 catalog_manager.cc:4841] TS 680b9950669e4d98abdc4b6ccf6fe0b4 (127.0.98.1:45309): tablet a6969e5953cc4a5f964df7d2b654eb95 (table test-table [id=92b9f26247bd4c789a117d60064da639]) successfully deleted
Jun 19 12:35:56 dist-test-slave-kdrb krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 12:35:56 dist-test-slave-kdrb krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:35:56 dist-test-slave-kdrb krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 12:35:56.416612 7959 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:35:56.409288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37168 (local address 127.0.98.62:42445)
0619 12:35:56.409453 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:56.409457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:56.409676 (+ 219us) server_negotiation.cc:407] Connection header received
0619 12:35:56.409812 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:56.409814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:56.409860 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:56.409952 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:56.410840 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.411373 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.412660 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.412893 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.412982 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:56.416484 (+ 3502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37168: BlockingRecv error: recv got EOF from 127.0.0.1:37168 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 12:35:56 dist-test-slave-kdrb krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:35:56.427067 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:56.418295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.0.98.62:42445)
0619 12:35:56.418418 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:56.418422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:56.418598 (+ 176us) server_negotiation.cc:407] Connection header received
0619 12:35:56.418759 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:56.418762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:56.418815 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:56.418927 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:56.419586 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.420107 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.420922 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.421145 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.421258 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:56.423751 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:56.423754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:56.423756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:56.423784 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:56.425288 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:56.425753 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:56.425756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:56.425758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:56.425835 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:56.426116 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:56.426121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:56.426123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:56.426316 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:56.426460 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:56.426751 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 12:35:56.426895 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 12:35:56.430503 7908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:35:56.431001 7908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:56.437544 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 56501a7327a441529afdc53a57e29b94 (DEFAULT_TABLE table=test-table [id=a0c73fd5ed0f4acfa787c2ac33575c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:56.437637 8046 tablet_service.cc:1467] Processing CreateTablet for tablet 56501a7327a441529afdc53a57e29b94 (DEFAULT_TABLE table=test-table [id=a0c73fd5ed0f4acfa787c2ac33575c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:56.437891 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56501a7327a441529afdc53a57e29b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:56.437891 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56501a7327a441529afdc53a57e29b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:56.439795 8185 tablet_service.cc:1467] Processing CreateTablet for tablet 56501a7327a441529afdc53a57e29b94 (DEFAULT_TABLE table=test-table [id=a0c73fd5ed0f4acfa787c2ac33575c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:56.440093 8185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56501a7327a441529afdc53a57e29b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:56.442597 8496 tablet_bootstrap.cc:492] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e: Bootstrap starting.
I20260619 12:35:56.443645 8496 tablet_bootstrap.cc:654] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:56.444603 8498 tablet_bootstrap.cc:492] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4: Bootstrap starting.
I20260619 12:35:56.445212 8495 tablet_bootstrap.cc:492] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f: Bootstrap starting.
I20260619 12:35:56.445518 8498 tablet_bootstrap.cc:654] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:56.446318 8495 tablet_bootstrap.cc:654] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:56.446799 8498 tablet_bootstrap.cc:492] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4: No bootstrap required, opened a new log
I20260619 12:35:56.446892 8498 ts_tablet_manager.cc:1397] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:35:56.447333 8498 raft_consensus.cc:348] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.447454 8498 raft_consensus.cc:374] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:56.447517 8498 raft_consensus.cc:729] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 680b9950669e4d98abdc4b6ccf6fe0b4, State: Initialized, Role: FOLLOWER
I20260619 12:35:56.447649 8498 consensus_queue.cc:260] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.447937 8498 ts_tablet_manager.cc:1428] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:35:56.449445 8496 tablet_bootstrap.cc:492] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e: No bootstrap required, opened a new log
I20260619 12:35:56.449544 8496 ts_tablet_manager.cc:1397] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260619 12:35:56.450011 8496 raft_consensus.cc:348] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.450134 8496 raft_consensus.cc:374] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:56.450169 8496 raft_consensus.cc:729] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d57187001f6464b9a5721b92d3b102e, State: Initialized, Role: FOLLOWER
I20260619 12:35:56.450302 8496 consensus_queue.cc:260] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.450585 8496 ts_tablet_manager.cc:1428] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:35:56.456341 8495 tablet_bootstrap.cc:492] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f: No bootstrap required, opened a new log
I20260619 12:35:56.456480 8495 ts_tablet_manager.cc:1397] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260619 12:35:56.456940 8495 raft_consensus.cc:348] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.457072 8495 raft_consensus.cc:374] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:56.457111 8495 raft_consensus.cc:729] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42896eb323d14d81b9137f2dd4954b3f, State: Initialized, Role: FOLLOWER
I20260619 12:35:56.457257 8495 consensus_queue.cc:260] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.457646 8495 ts_tablet_manager.cc:1428] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:35:56.757536 8427 raft_consensus.cc:482] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:56.757761 8427 raft_consensus.cc:504] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.758987 8427 leader_election.cc:288] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 680b9950669e4d98abdc4b6ccf6fe0b4 (127.0.98.1:45309), 9d57187001f6464b9a5721b92d3b102e (127.0.98.2:37345)
I20260619 12:35:56.763526 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:56.759251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45309 (local address 127.0.98.3:47477)
0619 12:35:56.759561 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:56.759578 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:56.759695 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:56.760152 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:56.760157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:56.760175 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:56.760289 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:56.760298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.761177 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.761184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:56.762401 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:56.762413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.763129 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.763132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:56.763195 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:56.763215 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 12:35:56.763275 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 12:35:56.763339 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":97,"threads_started":1}
I20260619 12:35:56.763537 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:56.759362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47477 (local address 127.0.98.1:45309)
0619 12:35:56.759552 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:56.759557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:56.759659 (+ 102us) server_negotiation.cc:407] Connection header received
0619 12:35:56.759848 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:56.759852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:56.759920 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:56.759993 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:56.760424 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.761058 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.762528 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.762998 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.763108 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:56.763176 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:56.763311 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 12:35:56.763386 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:35:56.764040 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56501a7327a441529afdc53a57e29b94" candidate_uuid: "42896eb323d14d81b9137f2dd4954b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" is_pre_election: true
I20260619 12:35:56.764214 8067 raft_consensus.cc:2454] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42896eb323d14d81b9137f2dd4954b3f in term 0.
I20260619 12:35:56.764313 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:56.759687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37345 (local address 127.0.98.3:53987)
0619 12:35:56.759960 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:56.759975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:56.760093 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:56.760393 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:56.760398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:56.760418 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:35:56.760552 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:56.760558 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.761896 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.761899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:56.762851 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:56.762858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.763895 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.763900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:56.763992 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:56.764009 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 12:35:56.764102 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 12:35:56.764173 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":70,"threads_started":1}
I20260619 12:35:56.764315 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:56.759840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53987 (local address 127.0.98.2:37345)
0619 12:35:56.760112 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:56.760117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:56.760132 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:35:56.760180 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:56.760183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:56.760225 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:56.760315 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:35:56.761098 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.761747 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.762977 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.763624 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.763750 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:56.763815 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:56.764119 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 12:35:56.764198 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
I20260619 12:35:56.764678 8280 leader_election.cc:302] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42896eb323d14d81b9137f2dd4954b3f, 680b9950669e4d98abdc4b6ccf6fe0b4; no voters:
I20260619 12:35:56.764765 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56501a7327a441529afdc53a57e29b94" candidate_uuid: "42896eb323d14d81b9137f2dd4954b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d57187001f6464b9a5721b92d3b102e" is_pre_election: true
I20260619 12:35:56.764932 8207 raft_consensus.cc:2454] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42896eb323d14d81b9137f2dd4954b3f in term 0.
I20260619 12:35:56.764945 8427 raft_consensus.cc:2781] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:56.765004 8427 raft_consensus.cc:482] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:56.765041 8427 raft_consensus.cc:3037] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:56.766067 8427 raft_consensus.cc:504] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.766373 8427 leader_election.cc:288] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [CANDIDATE]: Term 1 election: Requested vote from peers 680b9950669e4d98abdc4b6ccf6fe0b4 (127.0.98.1:45309), 9d57187001f6464b9a5721b92d3b102e (127.0.98.2:37345)
I20260619 12:35:56.766731 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56501a7327a441529afdc53a57e29b94" candidate_uuid: "42896eb323d14d81b9137f2dd4954b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4"
I20260619 12:35:56.766876 8067 raft_consensus.cc:3037] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:56.766845 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56501a7327a441529afdc53a57e29b94" candidate_uuid: "42896eb323d14d81b9137f2dd4954b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d57187001f6464b9a5721b92d3b102e"
I20260619 12:35:56.766960 8207 raft_consensus.cc:3037] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:56.768091 8207 raft_consensus.cc:2454] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42896eb323d14d81b9137f2dd4954b3f in term 1.
I20260619 12:35:56.768091 8067 raft_consensus.cc:2454] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42896eb323d14d81b9137f2dd4954b3f in term 1.
I20260619 12:35:56.768378 8280 leader_election.cc:302] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42896eb323d14d81b9137f2dd4954b3f, 9d57187001f6464b9a5721b92d3b102e; no voters:
I20260619 12:35:56.768553 8427 raft_consensus.cc:2781] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:56.768803 8427 raft_consensus.cc:686] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [term 1 LEADER]: Becoming Leader. State: Replica: 42896eb323d14d81b9137f2dd4954b3f, State: Running, Role: LEADER
I20260619 12:35:56.769068 8427 consensus_queue.cc:237] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.771680 7908 catalog_manager.cc:5495] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 42896eb323d14d81b9137f2dd4954b3f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "42896eb323d14d81b9137f2dd4954b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } health_report { overall_health: HEALTHY } } }
I20260619 12:35:56.819944 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:56.815911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55744 (local address 127.0.98.3:44967)
0619 12:35:56.816243 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:56.816250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:56.816264 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:35:56.816345 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:56.816347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:56.816403 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:56.816513 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:56.816801 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.817422 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.818328 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:56.818607 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:56.818702 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:56.818931 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:56.819434 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:56.819638 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:56.819669 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 12:35:56.819725 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":129,"threads_started":1}
I20260619 12:35:56.822607 8207 raft_consensus.cc:1261] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e [term 1 FOLLOWER]: Refusing update from remote peer 42896eb323d14d81b9137f2dd4954b3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:56.822672 8067 raft_consensus.cc:1261] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Refusing update from remote peer 42896eb323d14d81b9137f2dd4954b3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:56.823108 8504 consensus_queue.cc:1035] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:56.823333 8427 consensus_queue.cc:1035] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:56.834828 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:56.835035 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:56.837411 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359: Sending DeleteTablet for 3 replicas of tablet 56501a7327a441529afdc53a57e29b94
I20260619 12:35:56.838231 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet 56501a7327a441529afdc53a57e29b94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40274
I20260619 12:35:56.838428 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet 56501a7327a441529afdc53a57e29b94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55738
I20260619 12:35:56.838639 8481 tablet_replica.cc:330] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f: stopping tablet replica
I20260619 12:35:56.838649 8185 tablet_service.cc:1514] Processing DeleteTablet for tablet 56501a7327a441529afdc53a57e29b94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34240
I20260619 12:35:56.838775 8481 raft_consensus.cc:2229] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:35:56.838855 8482 tablet_replica.cc:330] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4: stopping tablet replica
I20260619 12:35:56.838985 8482 raft_consensus.cc:2229] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:56.838913 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 12:35:56.839112 8481 raft_consensus.cc:2258] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:56.839125 8482 raft_consensus.cc:2258] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:56.839313 8480 tablet_replica.cc:330] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e: stopping tablet replica
W20260619 12:35:56.839351 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:35:56.839433 8480 raft_consensus.cc:2229] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:56.839604 8480 raft_consensus.cc:2258] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:56.840662 8482 ts_tablet_manager.cc:1905] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:56.843362 8482 ts_tablet_manager.cc:1918] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:56.843441 8482 log.cc:1196] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-0/wal/wals/56501a7327a441529afdc53a57e29b94
I20260619 12:35:56.843986 8481 ts_tablet_manager.cc:1905] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:56.844293 8046 tablet_service.cc:1467] Processing CreateTablet for tablet 4356eb045d1e46fa850b4e82bce2c159 (DEFAULT_TABLE table=test-table [id=78e7267ed9474828b82a3d1745b645f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:56.844512 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4356eb045d1e46fa850b4e82bce2c159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:56.846333 8498 tablet_bootstrap.cc:492] T 4356eb045d1e46fa850b4e82bce2c159 P 680b9950669e4d98abdc4b6ccf6fe0b4: Bootstrap starting.
I20260619 12:35:56.846422 8481 ts_tablet_manager.cc:1918] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:56.846498 8481 log.cc:1196] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/56501a7327a441529afdc53a57e29b94
I20260619 12:35:56.846745 8481 ts_tablet_manager.cc:1939] T 56501a7327a441529afdc53a57e29b94 P 42896eb323d14d81b9137f2dd4954b3f: Deleting consensus metadata
I20260619 12:35:56.847218 8498 tablet_bootstrap.cc:654] T 4356eb045d1e46fa850b4e82bce2c159 P 680b9950669e4d98abdc4b6ccf6fe0b4: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:56.847275 8482 ts_tablet_manager.cc:1939] T 56501a7327a441529afdc53a57e29b94 P 680b9950669e4d98abdc4b6ccf6fe0b4: Deleting consensus metadata
I20260619 12:35:56.847282 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 4356eb045d1e46fa850b4e82bce2c159 (DEFAULT_TABLE table=test-table [id=78e7267ed9474828b82a3d1745b645f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:56.847556 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4356eb045d1e46fa850b4e82bce2c159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:56.848461 8480 ts_tablet_manager.cc:1905] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:56.848850 8498 tablet_bootstrap.cc:492] T 4356eb045d1e46fa850b4e82bce2c159 P 680b9950669e4d98abdc4b6ccf6fe0b4: No bootstrap required, opened a new log
I20260619 12:35:56.848929 8498 ts_tablet_manager.cc:1397] T 4356eb045d1e46fa850b4e82bce2c159 P 680b9950669e4d98abdc4b6ccf6fe0b4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 12:35:56.849407 8498 raft_consensus.cc:348] T 4356eb045d1e46fa850b4e82bce2c159 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.849524 8498 raft_consensus.cc:374] T 4356eb045d1e46fa850b4e82bce2c159 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:56.849555 8498 raft_consensus.cc:729] T 4356eb045d1e46fa850b4e82bce2c159 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 680b9950669e4d98abdc4b6ccf6fe0b4, State: Initialized, Role: FOLLOWER
I20260619 12:35:56.849663 8498 consensus_queue.cc:260] T 4356eb045d1e46fa850b4e82bce2c159 P 680b9950669e4d98abdc4b6ccf6fe0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.849908 8498 ts_tablet_manager.cc:1428] T 4356eb045d1e46fa850b4e82bce2c159 P 680b9950669e4d98abdc4b6ccf6fe0b4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 12:35:56.850447 8495 tablet_bootstrap.cc:492] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f: Bootstrap starting.
I20260619 12:35:56.850850 8480 ts_tablet_manager.cc:1918] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:35:56.850925 8480 log.cc:1196] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/56501a7327a441529afdc53a57e29b94
I20260619 12:35:56.851171 8480 ts_tablet_manager.cc:1939] T 56501a7327a441529afdc53a57e29b94 P 9d57187001f6464b9a5721b92d3b102e: Deleting consensus metadata
I20260619 12:35:56.851199 8495 tablet_bootstrap.cc:654] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:56.852376 8185 tablet_service.cc:1467] Processing CreateTablet for tablet 4356eb045d1e46fa850b4e82bce2c159 (DEFAULT_TABLE table=test-table [id=78e7267ed9474828b82a3d1745b645f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:35:56.852638 8185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4356eb045d1e46fa850b4e82bce2c159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:56.852938 8495 tablet_bootstrap.cc:492] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f: No bootstrap required, opened a new log
I20260619 12:35:56.853024 8495 ts_tablet_manager.cc:1397] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 12:35:56.853456 8495 raft_consensus.cc:348] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.853554 8495 raft_consensus.cc:374] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:56.853574 8495 raft_consensus.cc:729] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42896eb323d14d81b9137f2dd4954b3f, State: Initialized, Role: FOLLOWER
I20260619 12:35:56.853729 8495 consensus_queue.cc:260] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.854008 8495 ts_tablet_manager.cc:1428] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 12:35:56.854878 8496 tablet_bootstrap.cc:492] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e: Bootstrap starting.
I20260619 12:35:56.855847 8496 tablet_bootstrap.cc:654] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:56.856436 7896 catalog_manager.cc:4841] TS 9d57187001f6464b9a5721b92d3b102e (127.0.98.2:37345): tablet 56501a7327a441529afdc53a57e29b94 (table test-table [id=a0c73fd5ed0f4acfa787c2ac33575c17]) successfully deleted
I20260619 12:35:56.856901 7895 catalog_manager.cc:4841] TS 42896eb323d14d81b9137f2dd4954b3f (127.0.98.3:44967): tablet 56501a7327a441529afdc53a57e29b94 (table test-table [id=a0c73fd5ed0f4acfa787c2ac33575c17]) successfully deleted
I20260619 12:35:56.857084 7896 catalog_manager.cc:4841] TS 680b9950669e4d98abdc4b6ccf6fe0b4 (127.0.98.1:45309): tablet 56501a7327a441529afdc53a57e29b94 (table test-table [id=a0c73fd5ed0f4acfa787c2ac33575c17]) successfully deleted
I20260619 12:35:56.857863 8496 tablet_bootstrap.cc:492] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e: No bootstrap required, opened a new log
I20260619 12:35:56.857975 8496 ts_tablet_manager.cc:1397] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:35:56.858484 8496 raft_consensus.cc:348] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.858620 8496 raft_consensus.cc:374] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:56.858664 8496 raft_consensus.cc:729] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d57187001f6464b9a5721b92d3b102e, State: Initialized, Role: FOLLOWER
I20260619 12:35:56.858798 8496 consensus_queue.cc:260] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:56.859091 8496 ts_tablet_manager.cc:1428] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 12:35:57.060792 8442 raft_consensus.cc:482] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:35:57.060947 8442 raft_consensus.cc:504] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:57.061347 8442 leader_election.cc:288] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 680b9950669e4d98abdc4b6ccf6fe0b4 (127.0.98.1:45309), 42896eb323d14d81b9137f2dd4954b3f (127.0.98.3:44967)
I20260619 12:35:57.061892 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4356eb045d1e46fa850b4e82bce2c159" candidate_uuid: "9d57187001f6464b9a5721b92d3b102e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" is_pre_election: true
I20260619 12:35:57.061925 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4356eb045d1e46fa850b4e82bce2c159" candidate_uuid: "9d57187001f6464b9a5721b92d3b102e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42896eb323d14d81b9137f2dd4954b3f" is_pre_election: true
I20260619 12:35:57.062080 8067 raft_consensus.cc:2454] T 4356eb045d1e46fa850b4e82bce2c159 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d57187001f6464b9a5721b92d3b102e in term 0.
I20260619 12:35:57.062088 8347 raft_consensus.cc:2454] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d57187001f6464b9a5721b92d3b102e in term 0.
I20260619 12:35:57.062448 8140 leader_election.cc:302] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 680b9950669e4d98abdc4b6ccf6fe0b4, 9d57187001f6464b9a5721b92d3b102e; no voters:
I20260619 12:35:57.062665 8442 raft_consensus.cc:2781] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:35:57.062731 8442 raft_consensus.cc:482] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:35:57.062808 8442 raft_consensus.cc:3037] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:57.063936 8442 raft_consensus.cc:504] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:57.064348 8442 leader_election.cc:288] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [CANDIDATE]: Term 1 election: Requested vote from peers 680b9950669e4d98abdc4b6ccf6fe0b4 (127.0.98.1:45309), 42896eb323d14d81b9137f2dd4954b3f (127.0.98.3:44967)
I20260619 12:35:57.064723 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4356eb045d1e46fa850b4e82bce2c159" candidate_uuid: "9d57187001f6464b9a5721b92d3b102e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4"
I20260619 12:35:57.064860 8067 raft_consensus.cc:3037] T 4356eb045d1e46fa850b4e82bce2c159 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:57.064930 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4356eb045d1e46fa850b4e82bce2c159" candidate_uuid: "9d57187001f6464b9a5721b92d3b102e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42896eb323d14d81b9137f2dd4954b3f"
I20260619 12:35:57.065042 8347 raft_consensus.cc:3037] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:57.065860 8067 raft_consensus.cc:2454] T 4356eb045d1e46fa850b4e82bce2c159 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d57187001f6464b9a5721b92d3b102e in term 1.
I20260619 12:35:57.065975 8347 raft_consensus.cc:2454] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d57187001f6464b9a5721b92d3b102e in term 1.
I20260619 12:35:57.066177 8140 leader_election.cc:302] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 680b9950669e4d98abdc4b6ccf6fe0b4, 9d57187001f6464b9a5721b92d3b102e; no voters:
I20260619 12:35:57.066368 8442 raft_consensus.cc:2781] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:57.066583 8442 raft_consensus.cc:686] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [term 1 LEADER]: Becoming Leader. State: Replica: 9d57187001f6464b9a5721b92d3b102e, State: Running, Role: LEADER
I20260619 12:35:57.066746 8442 consensus_queue.cc:237] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } }
I20260619 12:35:57.068543 7908 catalog_manager.cc:5495] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d57187001f6464b9a5721b92d3b102e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9d57187001f6464b9a5721b92d3b102e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d57187001f6464b9a5721b92d3b102e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20260619 12:35:57.105898 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:57.102506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.0.98.2:37345)
0619 12:35:57.102640 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:57.102646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:57.102678 (+ 32us) server_negotiation.cc:407] Connection header received
0619 12:35:57.102863 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:57.102866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:57.102922 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:57.102999 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:57.103294 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:57.103939 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:57.104829 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:57.105065 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:57.105177 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:57.105441 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:57.105522 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:57.105601 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:57.105696 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 12:35:57.105750 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 12:35:57.117122 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:57.113550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52223 (local address 127.0.98.1:45309)
0619 12:35:57.113730 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:57.113734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:57.113812 (+ 78us) server_negotiation.cc:407] Connection header received
0619 12:35:57.113956 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:57.113959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:57.114008 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:57.114086 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:57.114347 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:57.114885 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:57.115815 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:57.116037 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:57.116147 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:57.116590 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:57.116700 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:57.116767 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:57.116929 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 12:35:57.116984 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 12:35:57.117154 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:57.113376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45309 (local address 127.0.98.2:52223)
0619 12:35:57.113720 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:57.113734 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:57.113858 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:57.114102 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:57.114104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:57.114111 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:35:57.114216 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:57.114220 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:57.114985 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:57.114987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:57.115699 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:57.115709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:57.116230 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:57.116233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:57.116333 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:57.116384 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:35:57.116795 (+ 411us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:35:57.116803 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 12:35:57.116897 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 12:35:57.116975 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":145,"threads_started":1}
I20260619 12:35:57.123423 8067 raft_consensus.cc:1261] T 4356eb045d1e46fa850b4e82bce2c159 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Refusing update from remote peer 9d57187001f6464b9a5721b92d3b102e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:57.123415 8347 raft_consensus.cc:1261] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f [term 1 FOLLOWER]: Refusing update from remote peer 9d57187001f6464b9a5721b92d3b102e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:35:57.123840 8442 consensus_queue.cc:1035] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [LEADER]: Connected to new peer: Peer: permanent_uuid: "680b9950669e4d98abdc4b6ccf6fe0b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:57.124037 8514 consensus_queue.cc:1035] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [LEADER]: Connected to new peer: Peer: permanent_uuid: "42896eb323d14d81b9137f2dd4954b3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:35:57.140981 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:57.137592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42569 (local address 127.0.98.2:37345)
0619 12:35:57.137699 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:57.137704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:57.137716 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:35:57.137762 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:57.137765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:57.137820 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:57.137918 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:35:57.138169 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:57.138688 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:57.139585 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:57.139825 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:57.139973 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:57.140401 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:35:57.140522 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:35:57.140598 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:57.140778 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 12:35:57.140838 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 12:35:57.140982 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:57.137147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37345 (local address 127.0.98.1:42569)
0619 12:35:57.137414 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:57.137430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:35:57.137532 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:35:57.137932 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:35:57.137935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:35:57.137943 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:35:57.138049 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:57.138054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:57.138804 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:57.138809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:57.139433 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:35:57.139441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:35:57.139972 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:35:57.139976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:35:57.140067 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:57.140246 (+ 179us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:35:57.140636 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:35:57.140644 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 12:35:57.140767 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 12:35:57.140862 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
I20260619 12:35:57.163874 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:57.164028 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:35:57.166003 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d343ffe1e6f74f0bb0db9998777ef359: Sending DeleteTablet for 3 replicas of tablet 4356eb045d1e46fa850b4e82bce2c159
I20260619 12:35:57.166585 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet 4356eb045d1e46fa850b4e82bce2c159 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40274
I20260619 12:35:57.166714 8185 tablet_service.cc:1514] Processing DeleteTablet for tablet 4356eb045d1e46fa850b4e82bce2c159 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34240
I20260619 12:35:57.166759 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet 4356eb045d1e46fa850b4e82bce2c159 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:35:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55738
I20260619 12:35:57.166952 8482 tablet_replica.cc:330] T 4356eb045d1e46fa850b4e82bce2c159 P 680b9950669e4d98abdc4b6ccf6fe0b4: stopping tablet replica
I20260619 12:35:57.167009 8481 tablet_replica.cc:330] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f: stopping tablet replica
I20260619 12:35:57.167091 8482 raft_consensus.cc:2229] T 4356eb045d1e46fa850b4e82bce2c159 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:57.167215 8482 raft_consensus.cc:2258] T 4356eb045d1e46fa850b4e82bce2c159 P 680b9950669e4d98abdc4b6ccf6fe0b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:57.167232 8481 raft_consensus.cc:2229] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:35:57.167279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 7982
I20260619 12:35:57.167395 8481 raft_consensus.cc:2258] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:57.167585 8480 tablet_replica.cc:330] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e: stopping tablet replica
I20260619 12:35:57.167733 8480 raft_consensus.cc:2229] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:35:57.167940 8480 raft_consensus.cc:2258] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:35:57.169565 8481 ts_tablet_manager.cc:1905] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:57.169565 8480 ts_tablet_manager.cc:1905] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:35:57.172230 8480 ts_tablet_manager.cc:1918] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:35:57.172230 8481 ts_tablet_manager.cc:1918] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:35:57.172327 8481 log.cc:1196] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/4356eb045d1e46fa850b4e82bce2c159
I20260619 12:35:57.172322 8480 log.cc:1196] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/4356eb045d1e46fa850b4e82bce2c159
I20260619 12:35:57.172611 8480 ts_tablet_manager.cc:1939] T 4356eb045d1e46fa850b4e82bce2c159 P 9d57187001f6464b9a5721b92d3b102e: Deleting consensus metadata
I20260619 12:35:57.172632 8481 ts_tablet_manager.cc:1939] T 4356eb045d1e46fa850b4e82bce2c159 P 42896eb323d14d81b9137f2dd4954b3f: Deleting consensus metadata
I20260619 12:35:57.173424 7895 catalog_manager.cc:4841] TS 42896eb323d14d81b9137f2dd4954b3f (127.0.98.3:44967): tablet 4356eb045d1e46fa850b4e82bce2c159 (table test-table [id=78e7267ed9474828b82a3d1745b645f5]) successfully deleted
I20260619 12:35:57.173426 7896 catalog_manager.cc:4841] TS 9d57187001f6464b9a5721b92d3b102e (127.0.98.2:37345): tablet 4356eb045d1e46fa850b4e82bce2c159 (table test-table [id=78e7267ed9474828b82a3d1745b645f5]) successfully deleted
W20260619 12:35:57.176086 7896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45309 (error 108)
I20260619 12:35:57.176265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 8122
I20260619 12:35:57.177711 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:57.177304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45309 (local address 127.0.0.1:40310)
0619 12:35:57.177537 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 12:35:57.177622 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
W20260619 12:35:57.177851 7896 catalog_manager.cc:4584] TS 680b9950669e4d98abdc4b6ccf6fe0b4 (127.0.98.1:45309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4356eb045d1e46fa850b4e82bce2c159: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45309: connect: Connection refused (error 111)
I20260619 12:35:57.183995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 8262
I20260619 12:35:57.191113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 7878
2026-06-19T12:35:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3879 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:35:57 dist-test-slave-kdrb krb5kdc[8532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:35:57 dist-test-slave-kdrb krb5kdc[8532](info): set up 2 sockets
Jun 19 12:35:57 dist-test-slave-kdrb krb5kdc[8532](info): commencing operation
krb5kdc: starting...
W20260619 12:35:58.232407 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:35:58 dist-test-slave-kdrb krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:35:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:35:58.359254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38729
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:58.475055 8545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:58.475425 8545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:58.475517 8545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:58.479984 8545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:58.480067 8545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:58.480087 8545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:58.480104 8545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:58.480120 8545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:58.485838 8545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:58.486927 8545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:58.488059 8545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:58.497629 8551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:58.497737 8545 server_base.cc:1034] running on GCE node
W20260619 12:35:58.497699 8550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:58.497629 8553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:58.498600 8545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:58.499953 8545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:58.501127 8545 hybrid_clock.cc:648] HybridClock initialized: now 1781872558501112 us; error 36 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T12:35:58Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1325 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:35:58 dist-test-slave-kdrb krb5kdc[8558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:35:58 dist-test-slave-kdrb krb5kdc[8558](info): set up 2 sockets
Jun 19 12:35:58 dist-test-slave-kdrb krb5kdc[8558](info): commencing operation
krb5kdc: starting...
W20260619 12:35:59.568192 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:35:59 dist-test-slave-kdrb krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:35:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:35:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:35:59.699990 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37907
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:35:59.817319 8571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:35:59.817688 8571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:35:59.817735 8571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:35:59.822088 8571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:35:59.822175 8571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:35:59.822194 8571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:35:59.822211 8571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:35:59.822227 8571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:35:59.827920 8571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:35:59.829059 8571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:35:59.830133 8571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:35:59.839360 8576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:59.839378 8579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:35:59.839361 8577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:35:59.839720 8571 server_base.cc:1034] running on GCE node
I20260619 12:35:59.840510 8571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:35:59.841811 8571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:35:59.843025 8571 hybrid_clock.cc:648] HybridClock initialized: now 1781872559843006 us; error 42 us; skew 500 ppm
Jun 19 12:35:59 dist-test-slave-kdrb krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:35:59.846101 8571 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:35:59.847162 8571 webserver.cc:460] Webserver started at http://127.0.98.62:41735/ using document root <none> and password file <none>
I20260619 12:35:59.847874 8571 fs_manager.cc:359] Metadata directory not provided
I20260619 12:35:59.847983 8571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:35:59.848366 8571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:35:59.850714 8571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "9b086f7d411f461792c43896ade90f79"
format_stamp: "Formatted at 2026-06-19 12:35:59 on dist-test-slave-kdrb"
server_key: "40ce2d229c4766932d58456d29685d79"
server_key_iv: "a8fc30650121efd81d275f6674cea7b2"
server_key_version: "encryptionkey@0"
I20260619 12:35:59.851402 8571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "9b086f7d411f461792c43896ade90f79"
format_stamp: "Formatted at 2026-06-19 12:35:59 on dist-test-slave-kdrb"
server_key: "40ce2d229c4766932d58456d29685d79"
server_key_iv: "a8fc30650121efd81d275f6674cea7b2"
server_key_version: "encryptionkey@0"
I20260619 12:35:59.855736 8571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:35:59.858512 8586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:35:59.859994 8571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:35:59.860184 8571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "9b086f7d411f461792c43896ade90f79"
format_stamp: "Formatted at 2026-06-19 12:35:59 on dist-test-slave-kdrb"
server_key: "40ce2d229c4766932d58456d29685d79"
server_key_iv: "a8fc30650121efd81d275f6674cea7b2"
server_key_version: "encryptionkey@0"
I20260619 12:35:59.860321 8571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:35:59.878793 8571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:35:59.882493 8571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:35:59.882762 8571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:35:59.890763 8571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37907
I20260619 12:35:59.890777 8638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37907 every 8 connection(s)
I20260619 12:35:59.891980 8571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:35:59.895016 8639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:35:59.897185 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 8571
I20260619 12:35:59.897351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:35:59.897643 392 external_mini_cluster.cc:1428] Setting key 6ae40708b66d4cb907726f4703427753
I20260619 12:35:59.902274 8639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79: Bootstrap starting.
I20260619 12:35:59.905194 8639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79: Neither blocks nor log segments found. Creating new log.
I20260619 12:35:59.906169 8639 log.cc:824] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:35:59 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:35:59.908751 8639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79: No bootstrap required, opened a new log
I20260619 12:35:59.913800 8639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b086f7d411f461792c43896ade90f79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37907 } }
I20260619 12:35:59.914117 8639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:35:59.914167 8639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b086f7d411f461792c43896ade90f79, State: Initialized, Role: FOLLOWER
I20260619 12:35:59.914685 8639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b086f7d411f461792c43896ade90f79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37907 } }
I20260619 12:35:59.914850 8639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:35:59.914912 8639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:35:59.914944 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:35:59.899288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51624 (local address 127.0.98.62:37907)
0619 12:35:59.899843 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0619 12:35:59.899865 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 12:35:59.899918 (+ 53us) server_negotiation.cc:407] Connection header received
0619 12:35:59.901038 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:35:59.901066 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:35:59.901531 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:35:59.901945 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:35:59.902748 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:59.904302 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:59.905271 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:35:59.906000 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:35:59.906177 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:35:59.909194 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:35:59.909200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:35:59.909219 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:35:59.909254 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:35:59.911369 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:59.911909 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:59.911915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:59.911922 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:59.912074 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:35:59.912435 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:35:59.912438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:35:59.912440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:35:59.913028 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:35:59.913255 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 12:35:59.913562 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 12:35:59.913893 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":213,"threads_started":1}
I20260619 12:35:59.915020 8639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:35:59.916154 8639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b086f7d411f461792c43896ade90f79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37907 } }
I20260619 12:35:59.916546 8639 leader_election.cc:302] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b086f7d411f461792c43896ade90f79; no voters:
I20260619 12:35:59.916882 8639 leader_election.cc:288] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:35:59.917006 8644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:35:59.917238 8644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 1 LEADER]: Becoming Leader. State: Replica: 9b086f7d411f461792c43896ade90f79, State: Running, Role: LEADER
I20260619 12:35:59.917567 8644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b086f7d411f461792c43896ade90f79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37907 } }
I20260619 12:35:59.918197 8639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:35:59.918815 8646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b086f7d411f461792c43896ade90f79. Latest consensus state: current_term: 1 leader_uuid: "9b086f7d411f461792c43896ade90f79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b086f7d411f461792c43896ade90f79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37907 } } }
I20260619 12:35:59.918962 8646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:59.919260 8645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b086f7d411f461792c43896ade90f79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b086f7d411f461792c43896ade90f79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37907 } } }
I20260619 12:35:59.919353 8645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [sys.catalog]: This master's current role is: LEADER
I20260619 12:35:59.919420 8651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:35:59.922816 8651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:35:59.928923 8651 catalog_manager.cc:1347] Generated new cluster ID: b24d11b6f5754a5b9ad14e3ec77f827e
I20260619 12:35:59.929020 8651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:35:59.950016 8651 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:35:59.951390 8651 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:35:59.960528 8651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79: Generated new TSK 0
I20260619 12:35:59.961345 8651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:35:59.999188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37907
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:36:00.116071 8665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:00.116415 8665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:00.116514 8665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:00.121168 8665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:00.121279 8665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:00.121373 8665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:36:00.127099 8665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:00.128355 8665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:00.129436 8665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:00.139779 8670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:00.139779 8671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:00.139825 8665 server_base.cc:1034] running on GCE node
W20260619 12:36:00.139745 8673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:00.140677 8665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:00.141440 8665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:00.142644 8665 hybrid_clock.cc:648] HybridClock initialized: now 1781872560142626 us; error 36 us; skew 500 ppm
Jun 19 12:36:00 dist-test-slave-kdrb krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:36:00.145673 8665 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:36:00.146662 8665 webserver.cc:460] Webserver started at http://127.0.98.1:42109/ using document root <none> and password file <none>
I20260619 12:36:00.147274 8665 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:00.147369 8665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:00.147657 8665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:36:00.149853 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "934d4ccba6084a339a6a5025a171fe66"
format_stamp: "Formatted at 2026-06-19 12:36:00 on dist-test-slave-kdrb"
server_key: "3891546db7e413415b2c21917b0213bc"
server_key_iv: "505c30ac6678a8425ffba5a8ea95e525"
server_key_version: "encryptionkey@0"
I20260619 12:36:00.150450 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "934d4ccba6084a339a6a5025a171fe66"
format_stamp: "Formatted at 2026-06-19 12:36:00 on dist-test-slave-kdrb"
server_key: "3891546db7e413415b2c21917b0213bc"
server_key_iv: "505c30ac6678a8425ffba5a8ea95e525"
server_key_version: "encryptionkey@0"
I20260619 12:36:00.154497 8665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 12:36:00.157416 8680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:00.158736 8665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:36:00.158910 8665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "934d4ccba6084a339a6a5025a171fe66"
format_stamp: "Formatted at 2026-06-19 12:36:00 on dist-test-slave-kdrb"
server_key: "3891546db7e413415b2c21917b0213bc"
server_key_iv: "505c30ac6678a8425ffba5a8ea95e525"
server_key_version: "encryptionkey@0"
I20260619 12:36:00.159045 8665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:00.175304 8665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:00.179119 8665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:00.179350 8665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:00.180099 8665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:36:00.180989 8665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:36:00.181059 8665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:00.181136 8665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:36:00.181178 8665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:00.190529 8665 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44215
I20260619 12:36:00.190547 8793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44215 every 8 connection(s)
I20260619 12:36:00.191547 8665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:36:00.196614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 8665
I20260619 12:36:00.196789 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:36:00.197058 392 external_mini_cluster.cc:1428] Setting key 12bb7e479dce396b71060bbb51283996
Jun 19 12:36:00 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:36:00.207578 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:00.193576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46483 (local address 127.0.98.62:37907)
0619 12:36:00.193730 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:00.193737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:00.194590 (+ 853us) server_negotiation.cc:407] Connection header received
0619 12:36:00.196154 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:00.196157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:00.196213 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:00.196325 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:00.197736 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:00.198533 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:00.200397 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:00.200817 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:00.200936 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:00.203799 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:00.203803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:00.203805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:36:00.203828 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:00.205357 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:00.205930 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:00.205939 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:00.205940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:00.206014 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:00.206374 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:00.206380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:00.206382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:00.206571 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:00.206680 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:00.207236 (+ 556us) server_negotiation.cc:299] Negotiation successful
0619 12:36:00.207382 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 12:36:00.208567 8796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:00.193956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:46483)
0619 12:36:00.194429 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:00.194474 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:00.195925 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:00.196514 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:00.196521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:00.196996 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:00.197544 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:00.197581 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:00.198670 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:00.198673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:00.200239 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:00.200255 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:00.200969 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:00.200972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:00.201117 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:00.201763 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:00.201787 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:00.203611 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:36:00.205503 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:36:00.205509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:36:00.205527 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:36:00.205807 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:36:00.206116 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:36:00.206119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:36:00.206121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:36:00.206269 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:36:00.206678 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:36:00.206685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:36:00.206957 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 12:36:00.207183 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 12:36:00.207567 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:36:00.209873 8794 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37907
I20260619 12:36:00.210130 8794 heartbeater.cc:464] Registering TS with master...
I20260619 12:36:00.210755 8794 heartbeater.cc:511] Master 127.0.98.62:37907 requested a full tablet report, sending...
I20260619 12:36:00.212502 8603 ts_manager.cc:194] Registered new tserver with Master: 934d4ccba6084a339a6a5025a171fe66 (127.0.98.1:44215)
I20260619 12:36:00.213768 8603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:36:00.224474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37907
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:36:00.340967 8799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:00.341315 8799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:00.341403 8799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:00.345753 8799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:00.345862 8799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:00.345954 8799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:36:00.351643 8799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:00.352802 8799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:00.353886 8799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:00.364147 8807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:00.364285 8804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:00.364360 8805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:00.364480 8799 server_base.cc:1034] running on GCE node
I20260619 12:36:00.364938 8799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:00.365672 8799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:00.366839 8799 hybrid_clock.cc:648] HybridClock initialized: now 1781872560366821 us; error 32 us; skew 500 ppm
Jun 19 12:36:00 dist-test-slave-kdrb krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:36:00.369961 8799 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:36:00.371018 8799 webserver.cc:460] Webserver started at http://127.0.98.2:45097/ using document root <none> and password file <none>
I20260619 12:36:00.371680 8799 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:00.371749 8799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:00.371992 8799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:36:00.374135 8799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "48da4f85fd20408981e8e87e5a111e27"
format_stamp: "Formatted at 2026-06-19 12:36:00 on dist-test-slave-kdrb"
server_key: "8f654e2d5b5b01f4ed14da57572df1ee"
server_key_iv: "ab24badfcea12d912f16caadad34cff4"
server_key_version: "encryptionkey@0"
I20260619 12:36:00.374783 8799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "48da4f85fd20408981e8e87e5a111e27"
format_stamp: "Formatted at 2026-06-19 12:36:00 on dist-test-slave-kdrb"
server_key: "8f654e2d5b5b01f4ed14da57572df1ee"
server_key_iv: "ab24badfcea12d912f16caadad34cff4"
server_key_version: "encryptionkey@0"
I20260619 12:36:00.378774 8799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 12:36:00.381492 8814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:00.382812 8799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:36:00.382956 8799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "48da4f85fd20408981e8e87e5a111e27"
format_stamp: "Formatted at 2026-06-19 12:36:00 on dist-test-slave-kdrb"
server_key: "8f654e2d5b5b01f4ed14da57572df1ee"
server_key_iv: "ab24badfcea12d912f16caadad34cff4"
server_key_version: "encryptionkey@0"
I20260619 12:36:00.383073 8799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:00.401284 8799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:00.404870 8799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:00.405102 8799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:00.405753 8799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:36:00.406620 8799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:36:00.406665 8799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:00.406728 8799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:36:00.406764 8799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:00.416378 8799 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45755
I20260619 12:36:00.416405 8927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45755 every 8 connection(s)
I20260619 12:36:00.417357 8799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:36:00.421159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 8799
I20260619 12:36:00.421307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:36:00.421655 392 external_mini_cluster.cc:1428] Setting key a54f640771712bdec73ef07d7d07dbc4
Jun 19 12:36:00 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:36:00.434322 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:00.419414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39525 (local address 127.0.98.62:37907)
0619 12:36:00.419580 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:00.419584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:00.420660 (+ 1076us) server_negotiation.cc:407] Connection header received
0619 12:36:00.422544 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:00.422548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:00.422601 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:00.422680 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:00.424398 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:00.424972 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:00.426794 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:00.427019 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:00.427103 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:00.430319 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:00.430323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:00.430326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:36:00.430353 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:00.431989 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:00.432567 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:00.432570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:00.432572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:00.432657 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:00.433087 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:00.433090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:00.433091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:00.433300 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:00.433391 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:00.434068 (+ 677us) server_negotiation.cc:299] Negotiation successful
0619 12:36:00.434188 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:36:00.435379 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:00.419895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:39525)
0619 12:36:00.420493 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:00.420549 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:00.422298 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:00.422895 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:00.422907 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:00.423462 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:00.424235 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:00.424258 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:00.425091 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:00.425097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:00.426654 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:00.426669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:00.427189 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:00.427196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:00.427452 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:00.428147 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:00.428174 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:00.430106 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:36:00.432111 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:36:00.432116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:36:00.432131 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:36:00.432425 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:36:00.432761 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:36:00.432767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:36:00.432771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:36:00.432957 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:36:00.433418 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:36:00.433423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:36:00.433693 (+ 270us) client_negotiation.cc:769] Sending connection context
0619 12:36:00.433965 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 12:36:00.434263 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":125,"threads_started":1}
I20260619 12:36:00.436724 8928 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37907
I20260619 12:36:00.437003 8928 heartbeater.cc:464] Registering TS with master...
I20260619 12:36:00.437542 8928 heartbeater.cc:511] Master 127.0.98.62:37907 requested a full tablet report, sending...
I20260619 12:36:00.438760 8603 ts_manager.cc:194] Registered new tserver with Master: 48da4f85fd20408981e8e87e5a111e27 (127.0.98.2:45755)
I20260619 12:36:00.439679 8603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:36:00.452689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37907
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:36:00.571406 8933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:00.571786 8933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:00.571835 8933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:00.576089 8933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:00.576169 8933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:00.576237 8933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:36:00.581991 8933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:00.583081 8933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:00.584177 8933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:00.594058 8939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:00.594203 8933 server_base.cc:1034] running on GCE node
W20260619 12:36:00.594089 8938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:00.594062 8941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:00.594733 8933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:00.595440 8933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:00.596685 8933 hybrid_clock.cc:648] HybridClock initialized: now 1781872560596669 us; error 30 us; skew 500 ppm
Jun 19 12:36:00 dist-test-slave-kdrb krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:36:00.599587 8933 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:36:00.600534 8933 webserver.cc:460] Webserver started at http://127.0.98.3:35207/ using document root <none> and password file <none>
I20260619 12:36:00.601105 8933 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:00.601178 8933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:00.601395 8933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:36:00.603550 8933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "f41c9157d00d4f47ad8c085310f19946"
format_stamp: "Formatted at 2026-06-19 12:36:00 on dist-test-slave-kdrb"
server_key: "77d39e61d80058bdd02e2b64421a76a4"
server_key_iv: "ad075af5252ed3e684e2504c6e19b0e8"
server_key_version: "encryptionkey@0"
I20260619 12:36:00.604151 8933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f41c9157d00d4f47ad8c085310f19946"
format_stamp: "Formatted at 2026-06-19 12:36:00 on dist-test-slave-kdrb"
server_key: "77d39e61d80058bdd02e2b64421a76a4"
server_key_iv: "ad075af5252ed3e684e2504c6e19b0e8"
server_key_version: "encryptionkey@0"
I20260619 12:36:00.608006 8933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:36:00.610659 8948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:00.611972 8933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:36:00.612113 8933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "f41c9157d00d4f47ad8c085310f19946"
format_stamp: "Formatted at 2026-06-19 12:36:00 on dist-test-slave-kdrb"
server_key: "77d39e61d80058bdd02e2b64421a76a4"
server_key_iv: "ad075af5252ed3e684e2504c6e19b0e8"
server_key_version: "encryptionkey@0"
I20260619 12:36:00.612236 8933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:00.624576 8933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:00.627723 8933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:00.627947 8933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:00.628552 8933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:36:00.629382 8933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:36:00.629456 8933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:00.629529 8933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:36:00.629580 8933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:00.638679 8933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39265
I20260619 12:36:00.638700 9061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39265 every 8 connection(s)
I20260619 12:36:00.639704 8933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 12:36:00 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:36:00.649585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 8933
I20260619 12:36:00.649726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:36:00.650048 392 external_mini_cluster.cc:1428] Setting key 5df9b44bf22a7297fa04014e68305c8e
I20260619 12:36:00.654433 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:00.641578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60517 (local address 127.0.98.62:37907)
0619 12:36:00.641713 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:00.641717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:00.642591 (+ 874us) server_negotiation.cc:407] Connection header received
0619 12:36:00.644087 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:00.644090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:00.644142 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:00.644226 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:00.645464 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:00.646005 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:00.647182 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:00.647391 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:00.647522 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:00.650388 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:00.650391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:00.650393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:36:00.650416 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:00.652616 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:00.653100 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:00.653103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:00.653104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:00.653177 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:00.653475 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:00.653477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:00.653478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:00.653646 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:00.653719 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:00.654201 (+ 482us) server_negotiation.cc:299] Negotiation successful
0619 12:36:00.654304 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 12:36:00.655287 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:00.641935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:60517)
0619 12:36:00.642418 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:00.642462 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:00.643846 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:00.644400 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:00.644407 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:00.644811 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:00.645309 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:00.645323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:00.646141 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:00.646144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:00.647070 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:00.647080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:00.647590 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:00.647593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:00.647750 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:00.648429 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:00.648446 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:00.650181 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:36:00.652717 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:36:00.652722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:36:00.652735 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:36:00.653015 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:36:00.653252 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:36:00.653254 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:36:00.653256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:36:00.653399 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:36:00.653728 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:36:00.653733 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:36:00.653986 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 12:36:00.654147 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 12:36:00.654427 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":115,"threads_started":1}
I20260619 12:36:00.656524 9062 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37907
I20260619 12:36:00.656849 9062 heartbeater.cc:464] Registering TS with master...
I20260619 12:36:00.657490 9062 heartbeater.cc:511] Master 127.0.98.62:37907 requested a full tablet report, sending...
I20260619 12:36:00.658597 8603 ts_manager.cc:194] Registered new tserver with Master: f41c9157d00d4f47ad8c085310f19946 (127.0.98.3:39265)
I20260619 12:36:00.659178 8603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60517
I20260619 12:36:00.664239 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 12:36:00.746340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 8665
I20260619 12:36:00.751616 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 8799
I20260619 12:36:00.757164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 8933
I20260619 12:36:00.762735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 8571
I20260619 12:36:00.769240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37907
--webserver_interface=127.0.98.62
--webserver_port=41735
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:36:00.881711 9071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:00.882015 9071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:00.882057 9071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:00.886150 9071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:36:00.886214 9071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:00.886234 9071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:00.886250 9071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:36:00.886265 9071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:36:00.891841 9071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:00.892943 9071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:00.893934 9071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:00.902812 9077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:00.902814 9079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:00.902814 9076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:00.903046 9071 server_base.cc:1034] running on GCE node
I20260619 12:36:00.903754 9071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:00.904874 9071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:00.906067 9071 hybrid_clock.cc:648] HybridClock initialized: now 1781872560906038 us; error 43 us; skew 500 ppm
Jun 19 12:36:00 dist-test-slave-kdrb krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:36:00.909018 9071 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:36:00.909914 9071 webserver.cc:460] Webserver started at http://127.0.98.62:41735/ using document root <none> and password file <none>
I20260619 12:36:00.910452 9071 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:00.910526 9071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:00.914784 9071 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 12:36:00.917166 9086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:00.918679 9071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 12:36:00.918841 9071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "9b086f7d411f461792c43896ade90f79"
format_stamp: "Formatted at 2026-06-19 12:35:59 on dist-test-slave-kdrb"
server_key: "40ce2d229c4766932d58456d29685d79"
server_key_iv: "a8fc30650121efd81d275f6674cea7b2"
server_key_version: "encryptionkey@0"
I20260619 12:36:00.919459 9071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:00.931725 9071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:00.934758 9071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:00.934979 9071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:00.942404 9071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37907
I20260619 12:36:00.942430 9138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37907 every 8 connection(s)
I20260619 12:36:00.943728 9071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:36:00.946029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 9071
I20260619 12:36:00.946535 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44215
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37907
--webserver_port=42109
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:36:00.948786 9139 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:36:00.950592 9139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79: Bootstrap starting.
I20260619 12:36:00.967530 9139 log.cc:824] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79: Log is configured to *not* fsync() on all Append() calls
I20260619 12:36:00.974365 9139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:36:00.974795 9139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79: Bootstrap complete.
I20260619 12:36:00.979027 9139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b086f7d411f461792c43896ade90f79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37907 } }
I20260619 12:36:00.979457 9139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b086f7d411f461792c43896ade90f79, State: Initialized, Role: FOLLOWER
I20260619 12:36:00.980068 9139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b086f7d411f461792c43896ade90f79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37907 } }
I20260619 12:36:00.980202 9139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:36:00.980248 9139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:36:00.980342 9139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:36:00.982457 9139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b086f7d411f461792c43896ade90f79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37907 } }
I20260619 12:36:00.982815 9139 leader_election.cc:302] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b086f7d411f461792c43896ade90f79; no voters:
I20260619 12:36:00.983181 9139 leader_election.cc:288] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 12:36:00.983247 9143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 12:36:00.983459 9143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [term 2 LEADER]: Becoming Leader. State: Replica: 9b086f7d411f461792c43896ade90f79, State: Running, Role: LEADER
I20260619 12:36:00.983865 9143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b086f7d411f461792c43896ade90f79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37907 } }
I20260619 12:36:00.984508 9139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:36:00.985419 9145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b086f7d411f461792c43896ade90f79. Latest consensus state: current_term: 2 leader_uuid: "9b086f7d411f461792c43896ade90f79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b086f7d411f461792c43896ade90f79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37907 } } }
I20260619 12:36:00.985541 9145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [sys.catalog]: This master's current role is: LEADER
I20260619 12:36:00.985821 9144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9b086f7d411f461792c43896ade90f79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b086f7d411f461792c43896ade90f79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37907 } } }
I20260619 12:36:00.985896 9144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79 [sys.catalog]: This master's current role is: LEADER
I20260619 12:36:00.989249 9158 catalog_manager.cc:1259] Loaded cluster ID: b24d11b6f5754a5b9ad14e3ec77f827e
I20260619 12:36:00.989342 9158 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79: loading cluster ID for follower catalog manager: success
I20260619 12:36:00.991225 9158 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79: acquiring CA information for follower catalog manager: success
I20260619 12:36:00.992449 9158 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:36:00.992604 9152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:36:00.993288 9152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:36:00.993700 9152 catalog_manager.cc:1259] Loaded cluster ID: b24d11b6f5754a5b9ad14e3ec77f827e
I20260619 12:36:00.993769 9152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:36:00.994292 9152 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:36:00.994686 9152 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9b086f7d411f461792c43896ade90f79: Loaded TSK: 0
I20260619 12:36:00.995314 9152 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:36:01.070173 9141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:01.070494 9141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:01.070583 9141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:01.074795 9141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:01.074898 9141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:01.074990 9141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:36:01.080361 9141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42109
--tserver_master_addrs=127.0.98.62:37907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:01.081453 9141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:01.082556 9141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:01.094110 9166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:01.094127 9165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:01.094488 9141 server_base.cc:1034] running on GCE node
W20260619 12:36:01.094537 9168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:01.094949 9141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:01.095805 9141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:01.097020 9141 hybrid_clock.cc:648] HybridClock initialized: now 1781872561096983 us; error 60 us; skew 500 ppm
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:36:01.100028 9141 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 12:36:01.101011 9141 webserver.cc:460] Webserver started at http://127.0.98.1:42109/ using document root <none> and password file <none>
I20260619 12:36:01.101599 9141 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:01.101707 9141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:01.105969 9141 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:36:01.108358 9175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:01.109965 9141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 12:36:01.110140 9141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "934d4ccba6084a339a6a5025a171fe66"
format_stamp: "Formatted at 2026-06-19 12:36:00 on dist-test-slave-kdrb"
server_key: "3891546db7e413415b2c21917b0213bc"
server_key_iv: "505c30ac6678a8425ffba5a8ea95e525"
server_key_version: "encryptionkey@0"
I20260619 12:36:01.110783 9141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:01.130923 9141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:01.134624 9141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:01.134855 9141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:01.135566 9141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:36:01.136449 9141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:36:01.136521 9141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:01.136595 9141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:36:01.136647 9141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:01.145936 9141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44215
I20260619 12:36:01.145958 9288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44215 every 8 connection(s)
I20260619 12:36:01.147061 9141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:36:01.153618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 9141
I20260619 12:36:01.154075 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45755
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37907
--webserver_port=45097
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 12:36:01.164057 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:01.149535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59345 (local address 127.0.98.62:37907)
0619 12:36:01.150125 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:01.150165 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:01.150402 (+ 237us) server_negotiation.cc:407] Connection header received
0619 12:36:01.152360 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:01.152373 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:01.152728 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:01.153111 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:01.154645 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.155832 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.157819 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.158532 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.158671 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.163224 (+ 4553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59345: BlockingRecv error: recv got EOF from 127.0.98.1:59345 (error 108)
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":147,"threads_started":1}
I20260619 12:36:01.164067 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:01.149634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:59345)
0619 12:36:01.150194 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:01.150265 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:01.151873 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:01.153191 (+ 1318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:01.153200 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:01.153776 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:01.154435 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.154455 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.156002 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.156013 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.157632 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.157663 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.158745 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.158754 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.159038 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.159829 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:01.159862 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:01.162984 (+ 3122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":136,"threads_started":1}
W20260619 12:36:01.164626 9289 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37907 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37907: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:01.170822 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:01.165163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:45991)
0619 12:36:01.165282 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:01.165298 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:01.165411 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:01.165701 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:01.165704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:01.165886 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:01.165984 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.165990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.166746 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.166751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.167315 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.167324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.167856 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.167860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.167965 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.168560 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:01.168572 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:01.170664 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 12:36:01.170919 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:01.165335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45991 (local address 127.0.98.62:37907)
0619 12:36:01.165489 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:01.165493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:01.165509 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:36:01.165551 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:01.165553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:01.165594 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:01.165688 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:01.166105 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.166630 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.167444 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.167721 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.167836 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.170806 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45991: BlockingRecv error: recv got EOF from 127.0.98.1:45991 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:01.178120 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:01.171549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:34227)
0619 12:36:01.171800 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:01.171813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:01.171903 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:01.172230 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:01.172232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:01.172421 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:01.172561 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.172567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.173798 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.173801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.174355 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.174363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.175104 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.175107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.175172 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.175791 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:01.175803 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:01.178001 (+ 2198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181}
W20260619 12:36:01.178409 9289 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:36:01.178527 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:01.171681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34227 (local address 127.0.98.62:37907)
0619 12:36:01.171813 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:01.171818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:01.171870 (+ 52us) server_negotiation.cc:407] Connection header received
0619 12:36:01.172003 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:01.172007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:01.172064 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:01.172170 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:01.172699 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.173631 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.174488 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.174957 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.175092 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.178220 (+ 3128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34227: BlockingRecv error: recv got EOF from 127.0.98.1:34227 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 12:36:01.279723 9293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:01.280038 9293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:01.280086 9293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:01.284225 9293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:01.284300 9293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:01.284363 9293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:36:01.290081 9293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45097
--tserver_master_addrs=127.0.98.62:37907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:01.291204 9293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:01.292256 9293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:01.302428 9298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:01.302429 9299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:01.302652 9293 server_base.cc:1034] running on GCE node
W20260619 12:36:01.303658 9301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:01.304628 9293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:01.305367 9293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:01.306573 9293 hybrid_clock.cc:648] HybridClock initialized: now 1781872561306534 us; error 58 us; skew 500 ppm
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:36:01.309554 9293 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 12:36:01.310524 9293 webserver.cc:460] Webserver started at http://127.0.98.2:45097/ using document root <none> and password file <none>
I20260619 12:36:01.311161 9293 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:01.311275 9293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:01.315629 9293 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:36:01.317960 9308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:01.319396 9293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:36:01.319581 9293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "48da4f85fd20408981e8e87e5a111e27"
format_stamp: "Formatted at 2026-06-19 12:36:00 on dist-test-slave-kdrb"
server_key: "8f654e2d5b5b01f4ed14da57572df1ee"
server_key_iv: "ab24badfcea12d912f16caadad34cff4"
server_key_version: "encryptionkey@0"
I20260619 12:36:01.320170 9293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:01.342980 9293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:01.346277 9293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:01.346496 9293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:01.347122 9293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:36:01.348006 9293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:36:01.348078 9293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:01.348150 9293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:36:01.348202 9293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:01.357693 9293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45755
I20260619 12:36:01.357721 9421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45755 every 8 connection(s)
I20260619 12:36:01.358847 9293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:36:01.361769 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 9293
I20260619 12:36:01.362234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39265
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37907
--webserver_port=35207
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 12:36:01.370599 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:01.360767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38297 (local address 127.0.98.62:37907)
0619 12:36:01.360931 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:01.360934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:01.361773 (+ 839us) server_negotiation.cc:407] Connection header received
0619 12:36:01.363304 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:01.363307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:01.363368 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:01.363479 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:01.364705 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.365236 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.366398 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.366617 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.366732 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.370487 (+ 3755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38297: BlockingRecv error: recv got EOF from 127.0.98.2:38297 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 12:36:01.371660 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:01.361160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:38297)
0619 12:36:01.361617 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:01.361658 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:01.363075 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:01.363687 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:01.363694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:01.364059 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:01.364556 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.364569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.365371 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.365382 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.366260 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.366270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.366773 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.366777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.366947 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.367595 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:01.367612 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:01.370359 (+ 2747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
W20260619 12:36:01.372205 9422 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37907 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37907: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:01.378458 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:01.372707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:49529)
0619 12:36:01.372936 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:01.372949 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:01.373030 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:01.373293 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:01.373295 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:01.373440 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:01.373535 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.373540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.374575 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.374578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.375126 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.375134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.375772 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.375775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.375877 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.376414 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:01.376425 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:01.378275 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 12:36:01.378690 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:01.372796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49529 (local address 127.0.98.62:37907)
0619 12:36:01.372907 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:01.372911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:01.373013 (+ 102us) server_negotiation.cc:407] Connection header received
0619 12:36:01.373118 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:01.373121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:01.373174 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:01.373272 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:01.373643 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.374464 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.375240 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.375621 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.375735 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.378430 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49529: BlockingRecv error: recv got EOF from 127.0.98.2:49529 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:01.384518 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:01.379119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:41609)
0619 12:36:01.379237 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:01.379253 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:01.379331 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:01.379573 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:01.379575 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:01.379706 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:01.379780 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.379784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.380806 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.380808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.381334 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.381341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.381905 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.381907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.381964 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.382472 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:01.382483 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:01.384376 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 12:36:01.384753 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:01.379190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41609 (local address 127.0.98.62:37907)
0619 12:36:01.379319 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:01.379323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:01.379338 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:36:01.379400 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:01.379404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:01.379446 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:01.379561 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:01.379901 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.380701 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.381450 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.381794 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.381896 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.384500 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41609: BlockingRecv error: recv got EOF from 127.0.98.2:41609 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 12:36:01.384760 9422 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:36:01.480741 9425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:01.481158 9425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:01.481240 9425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:01.485731 9425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:01.485822 9425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:01.485926 9425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:36:01.491822 9425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35207
--tserver_master_addrs=127.0.98.62:37907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:01.492980 9425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:01.494033 9425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:01.504316 9430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:01.504298 9433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:01.504335 9431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:01.504467 9425 server_base.cc:1034] running on GCE node
I20260619 12:36:01.504920 9425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:01.505569 9425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:01.506742 9425 hybrid_clock.cc:648] HybridClock initialized: now 1781872561506711 us; error 29 us; skew 500 ppm
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:36:01.509533 9425 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 12:36:01.510412 9425 webserver.cc:460] Webserver started at http://127.0.98.3:35207/ using document root <none> and password file <none>
I20260619 12:36:01.511008 9425 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:01.511086 9425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:01.515095 9425 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:36:01.517257 9440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:01.518469 9425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:36:01.518599 9425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "f41c9157d00d4f47ad8c085310f19946"
format_stamp: "Formatted at 2026-06-19 12:36:00 on dist-test-slave-kdrb"
server_key: "77d39e61d80058bdd02e2b64421a76a4"
server_key_iv: "ad075af5252ed3e684e2504c6e19b0e8"
server_key_version: "encryptionkey@0"
I20260619 12:36:01.519162 9425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:01.540912 9425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:01.544310 9425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:01.544526 9425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:01.545159 9425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:36:01.546011 9425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:36:01.546082 9425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:01.546155 9425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:36:01.546198 9425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:01.555435 9425 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39265
I20260619 12:36:01.555486 9553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39265 every 8 connection(s)
I20260619 12:36:01.556613 9425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:36:01.558929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 9425
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:01.569541 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:01.560023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51636 (local address 127.0.98.62:37907)
0619 12:36:01.560272 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:01.560277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:01.560290 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:36:01.560334 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:01.560337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:01.560381 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:01.560483 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:01.561195 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.562009 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.563276 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.563642 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.563723 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.565255 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:01.565262 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:01.565281 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:36:01.565316 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:01.567197 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:01.567739 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:01.567748 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:01.567757 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:01.567894 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:01.568206 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:01.568212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:01.568214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:01.568669 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:01.568819 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:01.569127 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 12:36:01.569368 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":116,"threads_started":1}
W20260619 12:36:01.569648 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:01.558503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51619 (local address 127.0.98.62:37907)
0619 12:36:01.558658 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:01.558663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:01.559758 (+ 1095us) server_negotiation.cc:407] Connection header received
0619 12:36:01.561711 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:01.561713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:01.561753 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:01.561813 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:01.563412 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.564118 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.565428 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.565640 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.565782 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.569515 (+ 3733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51619: BlockingRecv error: recv got EOF from 127.0.98.3:51619 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:36:01.570312 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:01.559026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:51619)
0619 12:36:01.559599 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:01.559656 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:01.561484 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:01.562000 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:01.562009 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:01.562551 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:01.563262 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.563280 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.564235 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.564242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.565293 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.565304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.565841 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.565845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.566035 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.566689 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:01.566705 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:01.569422 (+ 2717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":150,"threads_started":1}
W20260619 12:36:01.571592 9554 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37907 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37907: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:01.576856 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:01.572019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:50679)
0619 12:36:01.572199 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:01.572214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:01.572288 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:01.572550 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:01.572553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:01.572702 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:01.572796 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.572801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.573471 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.573473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.574005 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.574012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.574382 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.574385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.574451 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.574955 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:01.574967 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:01.576759 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113}
W20260619 12:36:01.577055 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:01.572112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50679 (local address 127.0.98.62:37907)
0619 12:36:01.572233 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:01.572237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:01.572252 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:36:01.572357 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:01.572360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:01.572415 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:01.572498 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:01.572895 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.573387 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.574092 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.574296 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.574378 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.576920 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50679: BlockingRecv error: recv got EOF from 127.0.98.3:50679 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:01 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:01.582015 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:01.577406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:48827)
0619 12:36:01.577536 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:01.577567 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:01.577644 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:01.577859 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:01.577862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:01.578018 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:01.578084 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.578089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.578774 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.578779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.579301 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:01.579308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.579783 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.579786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:01.579863 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.580280 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:01.580291 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:01.581879 (+ 1588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 12:36:01.582119 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:01.577495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48827 (local address 127.0.98.62:37907)
0619 12:36:01.577651 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:01.577655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:01.577670 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:36:01.577716 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:01.577719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:01.577760 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:01.577843 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:01.578199 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.578670 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.579418 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:01.579678 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:01.579784 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:01.582010 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48827: BlockingRecv error: recv got EOF from 127.0.98.3:48827 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260619 12:36:01.582261 9554 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 12:36:02 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:02 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:02.185679 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:02.179154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:37021)
0619 12:36:02.179504 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:02.179519 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:02.179631 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:02.179911 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:02.179914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:02.180101 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:02.180204 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:02.180209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:02.181302 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:02.181307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:02.182076 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:02.182088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:02.182570 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:02.182573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:02.182659 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:02.183231 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:02.183243 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:02.185504 (+ 2261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":94,"threads_started":1}
W20260619 12:36:02.185866 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:02.179306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37021 (local address 127.0.98.62:37907)
0619 12:36:02.179581 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:02.179585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:02.179600 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:36:02.179731 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:02.179734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:02.179779 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:02.179887 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:02.180474 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:02.181150 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:02.182194 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:02.182445 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:02.182565 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:02.185716 (+ 3151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37021: BlockingRecv error: recv got EOF from 127.0.98.1:37021 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":69,"threads_started":1}
Jun 19 12:36:02 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:02 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:02.391379 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:02.385404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:49255)
0619 12:36:02.385742 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:02.385756 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:02.385896 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:02.386140 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:02.386142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:02.386360 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:02.386470 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:02.386475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:02.387300 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:02.387305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:02.387980 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:02.387989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:02.388456 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:02.388460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:02.388559 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:02.389132 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:02.389153 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:02.391237 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":89,"threads_started":1}
W20260619 12:36:02.391570 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:02.385501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49255 (local address 127.0.98.62:37907)
0619 12:36:02.385678 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:02.385683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:02.385836 (+ 153us) server_negotiation.cc:407] Connection header received
0619 12:36:02.385981 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:02.385984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:02.386034 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:02.386121 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:02.386610 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:02.387188 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:02.388106 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:02.388342 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:02.388453 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:02.391394 (+ 2941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49255: BlockingRecv error: recv got EOF from 127.0.98.2:49255 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 19 12:36:02 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:02 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:02.589097 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:02.583006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:52829)
0619 12:36:02.583329 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:02.583343 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:02.583449 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:02.583710 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:02.583712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:02.583894 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:02.583990 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:02.583996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:02.584781 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:02.584784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:02.585547 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:02.585556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:02.585968 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:02.585970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:02.586059 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:02.586563 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:02.586574 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:02.588918 (+ 2344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":87,"threads_started":1}
W20260619 12:36:02.589179 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:02.583091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52829 (local address 127.0.98.62:37907)
0619 12:36:02.583233 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:02.583237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:02.583415 (+ 178us) server_negotiation.cc:407] Connection header received
0619 12:36:02.583550 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:02.583552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:02.583599 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:02.583672 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:02.584094 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:02.584687 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:02.585646 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:02.585863 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:02.585944 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:02.589060 (+ 3116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52829: BlockingRecv error: recv got EOF from 127.0.98.3:52829 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 12:36:03 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:03 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:03.192481 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:03.186493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:46145)
0619 12:36:03.186783 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:03.186849 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:03.186955 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:03.187226 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:03.187228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:03.187412 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:03.187575 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:03.187582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:03.188294 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:03.188296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:03.188859 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:03.188867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:03.189302 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:03.189305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:03.189380 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:03.189894 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:03.189908 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:03.192260 (+ 2352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
W20260619 12:36:03.192541 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:03.186579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46145 (local address 127.0.98.62:37907)
0619 12:36:03.186883 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:03.186888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:03.186912 (+ 24us) server_negotiation.cc:407] Connection header received
0619 12:36:03.187039 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:03.187041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:03.187100 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:03.187180 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:03.187676 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:03.188189 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:03.188960 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:03.189177 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:03.189256 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:03.192414 (+ 3158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46145: BlockingRecv error: recv got EOF from 127.0.98.1:46145 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":120,"threads_started":1}
Jun 19 12:36:03 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:03 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:03.398633 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:03.392329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:51149)
0619 12:36:03.392745 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:03.392767 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:03.392908 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:03.393195 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:03.393199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:03.393416 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:03.393509 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:03.393515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:03.394287 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:03.394292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:03.394871 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:03.394879 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:03.395356 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:03.395360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:03.395448 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:03.396230 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:03.396249 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:03.398492 (+ 2243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":101,"threads_started":1}
W20260619 12:36:03.398897 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:03.392413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51149 (local address 127.0.98.62:37907)
0619 12:36:03.392707 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:03.392712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:03.392867 (+ 155us) server_negotiation.cc:407] Connection header received
0619 12:36:03.393021 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:03.393024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:03.393078 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:03.393171 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:03.393634 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:03.394170 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:03.395000 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:03.395231 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:03.395348 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:03.398760 (+ 3412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51149: BlockingRecv error: recv got EOF from 127.0.98.2:51149 (error 108)
Metrics: {"server-negotiator.queue_time_us":191}
Jun 19 12:36:03 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:03 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:03.599888 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:03.589882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:58999)
0619 12:36:03.590153 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:03.590166 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:03.590254 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:03.590504 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:03.590506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:03.590682 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:03.590775 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:03.590780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:03.591552 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:03.591554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:03.592126 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:03.592134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:03.592560 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:03.592562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:03.592619 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:03.597608 (+ 4989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:03.597624 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:03.599677 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
W20260619 12:36:03.599921 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:03.590093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58999 (local address 127.0.98.62:37907)
0619 12:36:03.590222 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:03.590225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:03.590239 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:36:03.590345 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:03.590347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:03.590390 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:03.590462 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:03.590876 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:03.591399 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:03.592226 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:03.592464 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:03.592543 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:03.599815 (+ 7272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58999: BlockingRecv error: recv got EOF from 127.0.98.3:58999 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 12:36:04 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:04 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:04.199115 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:04.193462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:49951)
0619 12:36:04.193759 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:04.193777 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:04.193887 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:04.194173 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:04.194176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:04.194399 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:04.194493 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:04.194499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:04.195278 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:04.195283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:04.195922 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:04.195930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:04.196395 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:04.196399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:04.196490 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:04.197067 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:04.197078 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:04.198971 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":90,"threads_started":1}
W20260619 12:36:04.199250 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:04.193508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49951 (local address 127.0.98.62:37907)
0619 12:36:04.193814 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:04.193821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:04.193837 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:36:04.193989 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:04.193992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:04.194048 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:04.194153 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:04.194617 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:04.195166 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:04.196042 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:04.196272 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:04.196377 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:04.199111 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49951: BlockingRecv error: recv got EOF from 127.0.98.1:49951 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":99,"threads_started":1}
Jun 19 12:36:04 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:04 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:04.405133 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:04.399449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:35749)
0619 12:36:04.399822 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:04.399836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:04.399939 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:04.400198 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:04.400200 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:04.400405 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:04.400499 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:04.400504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:04.401283 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:04.401285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:04.401843 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:04.401849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:04.402266 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:04.402269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:04.402333 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:04.402891 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:04.402905 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:04.404938 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":100,"threads_started":1}
W20260619 12:36:04.405210 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:04.399614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35749 (local address 127.0.98.62:37907)
0619 12:36:04.399777 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:04.399780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:04.399939 (+ 159us) server_negotiation.cc:407] Connection header received
0619 12:36:04.400015 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:04.400017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:04.400064 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:04.400183 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:04.400633 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:04.401186 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:04.401946 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:04.402160 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:04.402239 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:04.405097 (+ 2858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35749: BlockingRecv error: recv got EOF from 127.0.98.2:35749 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 12:36:04 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:04 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:04.606307 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:04.600720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:46399)
0619 12:36:04.601162 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:04.601179 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:04.601298 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:04.601594 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:04.601597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:04.601802 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:04.601892 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:04.601897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:04.602655 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:04.602659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:04.603224 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:04.603231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:04.603763 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:04.603767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:04.603849 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:04.604329 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:04.604340 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:04.606115 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":80,"threads_started":1}
W20260619 12:36:04.606352 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:04.600781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46399 (local address 127.0.98.62:37907)
0619 12:36:04.600948 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:04.600953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:04.601243 (+ 290us) server_negotiation.cc:407] Connection header received
0619 12:36:04.601426 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:04.601429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:04.601485 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:04.601574 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:04.602009 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:04.602537 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:04.603339 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:04.603650 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:04.603760 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:04.606246 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46399: BlockingRecv error: recv got EOF from 127.0.98.3:46399 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 12:36:05 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:05 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:05.205739 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:05.200002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:43859)
0619 12:36:05.200279 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:05.200294 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:05.200384 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:05.200659 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:05.200661 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:05.200859 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:05.200953 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:05.200958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:05.201757 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:05.201760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:05.202338 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:05.202347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:05.202865 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:05.202869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:05.202936 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:05.203583 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:05.203595 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:05.205609 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
W20260619 12:36:05.206012 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:05.200208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43859 (local address 127.0.98.62:37907)
0619 12:36:05.200457 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:05.200461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:05.200472 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:36:05.200505 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:05.200507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:05.200547 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:05.200632 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:05.201075 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:05.201608 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:05.202467 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:05.202731 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:05.202814 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:05.205876 (+ 3062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43859: BlockingRecv error: recv got EOF from 127.0.98.1:43859 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":95,"threads_started":1}
Jun 19 12:36:05 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:05 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:05.411329 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:05.405986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:41211)
0619 12:36:05.406258 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:05.406273 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:05.406369 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:05.406587 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:05.406589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:05.406781 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:05.406876 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:05.406881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:05.407631 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:05.407634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:05.408223 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:05.408232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:05.408641 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:05.408644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:05.408703 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:05.409196 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:05.409208 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:05.411116 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":103,"threads_started":1}
W20260619 12:36:05.411370 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:05.406106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41211 (local address 127.0.98.62:37907)
0619 12:36:05.406257 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:05.406261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:05.406343 (+ 82us) server_negotiation.cc:407] Connection header received
0619 12:36:05.406439 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:05.406441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:05.406485 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:05.406556 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:05.406982 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:05.407540 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:05.408339 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:05.408549 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:05.408620 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:05.411257 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41211: BlockingRecv error: recv got EOF from 127.0.98.2:41211 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 12:36:05 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:05 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:05.612543 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:05.607252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:38757)
0619 12:36:05.607598 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:05.607612 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:05.607711 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:05.607945 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:05.607947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:05.608116 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:05.608208 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:05.608213 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:05.608929 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:05.608931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:05.609478 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:05.609485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:05.609896 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:05.609898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:05.609956 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:05.610429 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:05.610440 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:05.612425 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
W20260619 12:36:05.612789 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:05.607337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38757 (local address 127.0.98.62:37907)
0619 12:36:05.607512 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:05.607519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:05.607666 (+ 147us) server_negotiation.cc:407] Connection header received
0619 12:36:05.607777 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:05.607778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:05.607841 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:05.607907 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:05.608306 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:05.608838 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:05.609576 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:05.609788 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:05.609881 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:05.612606 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38757: BlockingRecv error: recv got EOF from 127.0.98.3:38757 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 12:36:06 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:06 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:06.212060 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:06.206638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:42039)
0619 12:36:06.206929 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:06.206942 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:06.207029 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:06.207279 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:06.207281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:06.207457 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:06.207625 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:06.207631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:06.208363 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:06.208366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:06.208933 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:06.208942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:06.209350 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:06.209352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:06.209411 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:06.210097 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:06.210110 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:06.211940 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":92,"threads_started":1}
W20260619 12:36:06.212306 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:06.206705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42039 (local address 127.0.98.62:37907)
0619 12:36:06.206987 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:06.206991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:06.207005 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:36:06.207121 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:06.207123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:06.207167 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:06.207244 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:06.207730 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:06.208264 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:06.209035 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:06.209251 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:06.209329 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:06.212146 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42039: BlockingRecv error: recv got EOF from 127.0.98.1:42039 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":63,"threads_started":1}
Jun 19 12:36:06 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:06 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:06.420855 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:06.412332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:50877)
0619 12:36:06.412624 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:06.412640 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:06.412736 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:06.413023 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:06.413026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:06.413234 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:06.413331 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:06.413337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:06.414106 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:06.414109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:06.414728 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:06.414737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:06.415247 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:06.415251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:06.415320 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:06.418738 (+ 3418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:06.418751 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:06.420703 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":128,"threads_started":1}
W20260619 12:36:06.421011 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:06.412537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50877 (local address 127.0.98.62:37907)
0619 12:36:06.412673 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:06.412682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:06.412697 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:36:06.412830 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:06.412832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:06.412876 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:06.412951 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:06.413463 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:06.413999 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:06.414872 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:06.415103 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:06.415187 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:06.420855 (+ 5668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50877: BlockingRecv error: recv got EOF from 127.0.98.2:50877 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 12:36:06 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:06 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:06.620456 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:06.613395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:36061)
0619 12:36:06.613780 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:06.613797 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:06.614039 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:06.614439 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:06.614442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:06.614655 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:06.614762 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:06.614768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:06.615975 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:06.615979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:06.616801 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:06.616821 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:06.617476 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:06.617479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:06.617554 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:06.618153 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:06.618168 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:06.620269 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":97,"threads_started":1}
W20260619 12:36:06.621055 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:06.613457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36061 (local address 127.0.98.62:37907)
0619 12:36:06.613612 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:06.613617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:06.613934 (+ 317us) server_negotiation.cc:407] Connection header received
0619 12:36:06.614158 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:06.614161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:06.614211 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:06.614274 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:06.614925 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:06.615840 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:06.616943 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:06.617259 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:06.617345 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:06.620400 (+ 3055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36061: BlockingRecv error: recv got EOF from 127.0.98.3:36061 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 12:36:07 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:07 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:07.218953 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:07.213042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:48969)
0619 12:36:07.213325 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:07.213339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:07.213436 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:07.213713 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:07.213715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:07.213897 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:07.213994 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:07.213999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:07.214842 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:07.214850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:07.215573 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:07.215589 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:07.216214 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:07.216217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:07.216283 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:07.216894 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:07.216908 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:07.218784 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":94,"threads_started":1}
W20260619 12:36:07.219069 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:07.213132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48969 (local address 127.0.98.62:37907)
0619 12:36:07.213417 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:07.213420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:07.213436 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:36:07.213539 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:07.213542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:07.213584 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:07.213670 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:07.214096 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:07.214713 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:07.215705 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:07.216072 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:07.216172 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:07.218937 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48969: BlockingRecv error: recv got EOF from 127.0.98.1:48969 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":136,"threads_started":1}
Jun 19 12:36:07 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:07 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:07.427754 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:07.421737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:41993)
0619 12:36:07.422038 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:07.422058 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:07.422227 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:07.422518 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:07.422522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:07.422754 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:07.422884 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:07.422892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:07.423853 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:07.423857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:07.424503 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:07.424512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:07.425019 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:07.425022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:07.425107 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:07.425659 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:07.425670 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:07.427571 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
W20260619 12:36:07.427819 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:07.421832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41993 (local address 127.0.98.62:37907)
0619 12:36:07.421997 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:07.422001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:07.422163 (+ 162us) server_negotiation.cc:407] Connection header received
0619 12:36:07.422356 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:07.422359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:07.422406 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:07.422495 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:07.423014 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:07.423695 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:07.424635 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:07.424893 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:07.425016 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:07.427708 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41993: BlockingRecv error: recv got EOF from 127.0.98.2:41993 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 12:36:07 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:07 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:07.627398 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:07.621449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:53453)
0619 12:36:07.621744 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:07.621759 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:07.621906 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:07.622201 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:07.622204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:07.622441 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:07.622535 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:07.622541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:07.623331 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:07.623337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:07.624052 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:07.624062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:07.624633 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:07.624637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:07.624715 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:07.625225 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:07.625237 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:07.627207 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
W20260619 12:36:07.627461 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:07.621554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53453 (local address 127.0.98.62:37907)
0619 12:36:07.621720 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:07.621725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:07.621842 (+ 117us) server_negotiation.cc:407] Connection header received
0619 12:36:07.622012 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:07.622018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:07.622080 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:07.622166 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:07.622650 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:07.623218 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:07.624172 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:07.624521 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:07.624632 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:07.627349 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53453: BlockingRecv error: recv got EOF from 127.0.98.3:53453 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 12:36:08 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:08 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:08.225647 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:08.219891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:46425)
0619 12:36:08.220196 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:08.220209 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:08.220304 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:08.220537 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:08.220539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:08.220739 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:08.220839 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:08.220844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:08.221556 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:08.221559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:08.222233 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:08.222243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:08.222680 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:08.222682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:08.222744 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:08.223424 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:08.223437 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:08.225530 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":90,"threads_started":1}
W20260619 12:36:08.225883 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:08.219936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46425 (local address 127.0.98.62:37907)
0619 12:36:08.220181 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:08.220185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:08.220276 (+ 91us) server_negotiation.cc:407] Connection header received
0619 12:36:08.220382 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:08.220384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:08.220427 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:08.220502 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:08.220933 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:08.221449 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:08.222335 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:08.222558 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:08.222637 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:08.225743 (+ 3106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46425: BlockingRecv error: recv got EOF from 127.0.98.1:46425 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
Jun 19 12:36:08 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:08 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:08.434695 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:08.428818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:41967)
0619 12:36:08.429136 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:08.429149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:08.429255 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:08.429491 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:08.429493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:08.429670 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:08.429759 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:08.429764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:08.430517 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:08.430519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:08.431068 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:08.431075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:08.431673 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:08.431677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:08.431764 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:08.432429 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:08.432444 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:08.434575 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":97,"threads_started":1}
W20260619 12:36:08.434867 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:08.428905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41967 (local address 127.0.98.62:37907)
0619 12:36:08.429069 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:08.429077 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:08.429230 (+ 153us) server_negotiation.cc:407] Connection header received
0619 12:36:08.429341 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:08.429343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:08.429387 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:08.429459 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:08.429884 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:08.430423 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:08.431202 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:08.431536 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:08.431658 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:08.434726 (+ 3068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41967: BlockingRecv error: recv got EOF from 127.0.98.2:41967 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 12:36:08 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:08 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:08.638417 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:08.628418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:56453)
0619 12:36:08.628717 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:08.628731 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:08.628836 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:08.629109 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:08.629112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:08.629315 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:08.629438 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:08.629444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:08.630221 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:08.630224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:08.630806 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:08.630815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:08.631313 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:08.631315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:08.631384 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:08.636342 (+ 4958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:08.636356 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:08.638284 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
W20260619 12:36:08.638566 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:08.628498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56453 (local address 127.0.98.62:37907)
0619 12:36:08.628699 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:08.628703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:08.628827 (+ 124us) server_negotiation.cc:407] Connection header received
0619 12:36:08.628912 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:08.628915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:08.628963 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:08.629040 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:08.629553 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:08.630086 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:08.630964 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:08.631198 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:08.631287 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:08.638445 (+ 7158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56453: BlockingRecv error: recv got EOF from 127.0.98.3:56453 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 12:36:09 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:09 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:09.232451 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:09.226563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:41451)
0619 12:36:09.226921 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:09.226935 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:09.227044 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:09.227318 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:09.227320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:09.227564 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:09.227685 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:09.227691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:09.228482 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:09.228485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:09.229102 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:09.229110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:09.229587 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:09.229590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:09.229682 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:09.230261 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:09.230274 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:09.232314 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":123,"threads_started":1}
W20260619 12:36:09.232609 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:09.226715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41451 (local address 127.0.98.62:37907)
0619 12:36:09.226999 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:09.227003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:09.227017 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:36:09.227149 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:09.227152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:09.227196 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:09.227279 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:09.227819 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:09.228364 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:09.229234 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:09.229467 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:09.229582 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:09.232475 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41451: BlockingRecv error: recv got EOF from 127.0.98.1:41451 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":82,"threads_started":1}
Jun 19 12:36:09 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:09 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:09.441745 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:09.435592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:52831)
0619 12:36:09.435935 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:09.435948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:09.436093 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:09.436392 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:09.436394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:09.436607 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:09.436704 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:09.436709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:09.437637 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:09.437642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:09.438210 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:09.438217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:09.438723 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:09.438727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:09.438809 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:09.439371 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:09.439383 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:09.441615 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
W20260619 12:36:09.441946 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:09.435656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52831 (local address 127.0.98.62:37907)
0619 12:36:09.435854 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:09.435859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:09.436033 (+ 174us) server_negotiation.cc:407] Connection header received
0619 12:36:09.436198 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:09.436202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:09.436263 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:09.436355 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:09.436861 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:09.437512 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:09.438342 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:09.438601 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:09.438718 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:09.441802 (+ 3084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52831: BlockingRecv error: recv got EOF from 127.0.98.2:52831 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 12:36:09 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:09 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:09.645082 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:09.639367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:55753)
0619 12:36:09.639730 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:09.639747 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:09.639874 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:09.640233 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:09.640235 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:09.640424 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:09.640523 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:09.640528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:09.641316 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:09.641318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:09.641892 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:09.641900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:09.642328 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:09.642339 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:09.642403 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:09.643020 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:09.643035 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:09.644951 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":128,"threads_started":1}
W20260619 12:36:09.645280 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:09.639433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55753 (local address 127.0.98.62:37907)
0619 12:36:09.639590 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:09.639596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:09.639834 (+ 238us) server_negotiation.cc:407] Connection header received
0619 12:36:09.640027 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:09.640030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:09.640085 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:09.640161 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:09.640662 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:09.641203 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:09.642000 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:09.642223 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:09.642307 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:09.645122 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55753: BlockingRecv error: recv got EOF from 127.0.98.3:55753 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 12:36:10 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:10 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:10.239265 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:10.233513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:41437)
0619 12:36:10.233791 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:10.233805 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:10.233897 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:10.234144 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:10.234146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:10.234336 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:10.234431 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:10.234436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:10.235302 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:10.235306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:10.235938 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:10.235948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:10.236500 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:10.236503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:10.236563 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:10.237170 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:10.237198 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:10.239141 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":102,"threads_started":1}
W20260619 12:36:10.239476 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:10.233634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41437 (local address 127.0.98.62:37907)
0619 12:36:10.233871 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:10.233875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:10.233889 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:36:10.233978 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:10.233981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:10.234025 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:10.234107 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:10.234593 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:10.235161 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:10.236143 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:10.236384 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:10.236479 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:10.239315 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41437: BlockingRecv error: recv got EOF from 127.0.98.1:41437 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
Jun 19 12:36:10 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:10 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:10.448443 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:10.442673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:42731)
0619 12:36:10.442933 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:10.442948 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:10.443085 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:10.443369 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:10.443373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:10.443622 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:10.443714 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:10.443720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:10.444502 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:10.444506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:10.445078 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:10.445086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:10.445563 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:10.445566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:10.445650 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:10.446161 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:10.446172 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:10.448249 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":113,"threads_started":1}
W20260619 12:36:10.448505 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:10.442735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42731 (local address 127.0.98.62:37907)
0619 12:36:10.442873 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:10.442878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:10.443039 (+ 161us) server_negotiation.cc:407] Connection header received
0619 12:36:10.443196 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:10.443200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:10.443253 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:10.443342 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:10.443850 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:10.444386 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:10.445202 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:10.445439 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:10.445547 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:10.448396 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42731: BlockingRecv error: recv got EOF from 127.0.98.2:42731 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 12:36:10 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:10 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:10.651351 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:10.645978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:49861)
0619 12:36:10.646245 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:10.646259 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:10.646374 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:10.646608 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:10.646610 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:10.646788 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:10.646886 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:10.646891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:10.647664 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:10.647667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:10.648213 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:10.648221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:10.648644 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:10.648646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:10.648705 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:10.649221 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:10.649232 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:10.651219 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":98,"threads_started":1}
W20260619 12:36:10.651541 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:10.646036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49861 (local address 127.0.98.62:37907)
0619 12:36:10.646178 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:10.646183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:10.646335 (+ 152us) server_negotiation.cc:407] Connection header received
0619 12:36:10.646454 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:10.646456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:10.646504 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:10.646573 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:10.646987 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:10.647530 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:10.648329 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:10.648545 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:10.648624 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:10.651389 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49861: BlockingRecv error: recv got EOF from 127.0.98.3:49861 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 12:36:11 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:11 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:11.246349 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:11.240392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:34219)
0619 12:36:11.240908 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:11.240926 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:11.241044 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:11.241383 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:11.241386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:11.241593 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:11.241685 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:11.241691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:11.242467 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:11.242472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:11.243055 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:11.243064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:11.243571 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:11.243575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:11.243661 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:11.244193 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:11.244206 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:11.246160 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":130,"threads_started":1}
W20260619 12:36:11.246415 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:11.240529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34219 (local address 127.0.98.62:37907)
0619 12:36:11.240793 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:11.240796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:11.241009 (+ 213us) server_negotiation.cc:407] Connection header received
0619 12:36:11.241159 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:11.241162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:11.241223 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:11.241349 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:11.241816 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:11.242351 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:11.243190 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:11.243423 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:11.243564 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:11.246295 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34219: BlockingRecv error: recv got EOF from 127.0.98.1:34219 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":76,"threads_started":1}
Jun 19 12:36:11 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:11 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:11.455763 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:11.449393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:51593)
0619 12:36:11.449666 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:11.449680 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:11.449783 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:11.450097 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:11.450099 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:11.450300 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:11.450402 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:11.450408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:11.451278 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:11.451281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:11.452003 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:11.452015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:11.452552 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:11.452556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:11.452628 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:11.453302 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:11.453314 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:11.455594 (+ 2280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":97,"threads_started":1}
W20260619 12:36:11.455891 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:11.449491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51593 (local address 127.0.98.62:37907)
0619 12:36:11.449643 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:11.449646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:11.449799 (+ 153us) server_negotiation.cc:407] Connection header received
0619 12:36:11.449857 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:11.449859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:11.449912 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:11.450007 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:11.450567 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:11.451118 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:11.452162 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:11.452418 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:11.452514 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:11.455757 (+ 3243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51593: BlockingRecv error: recv got EOF from 127.0.98.2:51593 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 12:36:11 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:11 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:11.658123 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:11.652348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:37251)
0619 12:36:11.652699 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:11.652715 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:11.652840 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:11.653153 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:11.653157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:11.653338 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:11.653432 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:11.653438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:11.654238 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:11.654241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:11.654801 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:11.654809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:11.655301 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:11.655303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:11.655373 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:11.655908 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:11.655919 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:11.657995 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
W20260619 12:36:11.658291 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:11.652461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37251 (local address 127.0.98.62:37907)
0619 12:36:11.652598 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:11.652602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:11.652818 (+ 216us) server_negotiation.cc:407] Connection header received
0619 12:36:11.652949 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:11.652953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:11.653011 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:11.653088 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:11.653584 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:11.654116 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:11.654948 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:11.655189 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:11.655278 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:11.658189 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37251: BlockingRecv error: recv got EOF from 127.0.98.3:37251 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 12:36:12 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:12 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:12.252561 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:12.247184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:40387)
0619 12:36:12.247443 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:12.247457 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:12.247620 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:12.247882 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:12.247884 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:12.248059 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:12.248155 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:12.248160 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:12.248894 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:12.248897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:12.249452 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:12.249459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:12.249926 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:12.249928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:12.249985 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:12.250515 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:12.250527 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:12.252415 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
W20260619 12:36:12.252725 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:12.247253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40387 (local address 127.0.98.62:37907)
0619 12:36:12.247524 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:12.247528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:12.247600 (+ 72us) server_negotiation.cc:407] Connection header received
0619 12:36:12.247704 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:12.247706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:12.247750 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:12.247834 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:12.248260 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:12.248796 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:12.249590 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:12.249821 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:12.249910 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:12.252584 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40387: BlockingRecv error: recv got EOF from 127.0.98.1:40387 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
Jun 19 12:36:12 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:12 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:12.462719 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:12.456641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:37237)
0619 12:36:12.457078 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:12.457096 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:12.457250 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:12.457528 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:12.457531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:12.457742 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:12.457839 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:12.457845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:12.458723 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:12.458729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:12.459620 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:12.459632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:12.460218 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:12.460221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:12.460284 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:12.460779 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:12.460791 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:12.462599 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":88,"threads_started":1}
W20260619 12:36:12.462894 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:12.456888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37237 (local address 127.0.98.62:37907)
0619 12:36:12.457057 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:12.457062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:12.457186 (+ 124us) server_negotiation.cc:407] Connection header received
0619 12:36:12.457362 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:12.457365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:12.457420 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:12.457512 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:12.457965 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:12.458612 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:12.459759 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:12.460110 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:12.460196 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:12.462737 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37237: BlockingRecv error: recv got EOF from 127.0.98.2:37237 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 12:36:12 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:12 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:12.665126 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:12.659167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:39921)
0619 12:36:12.659446 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:12.659461 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:12.659623 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:12.659949 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:12.659952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:12.660139 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:12.660254 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:12.660260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:12.661120 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:12.661123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:12.661762 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:12.661773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:12.662316 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:12.662319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:12.662388 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:12.663020 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:12.663033 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:12.664985 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":101,"threads_started":1}
W20260619 12:36:12.665259 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:12.659355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39921 (local address 127.0.98.62:37907)
0619 12:36:12.659523 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:12.659527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:12.659570 (+ 43us) server_negotiation.cc:407] Connection header received
0619 12:36:12.659710 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:12.659712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:12.659759 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:12.659869 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:12.660396 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:12.660971 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:12.661882 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:12.662196 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:12.662313 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:12.665139 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39921: BlockingRecv error: recv got EOF from 127.0.98.3:39921 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 12:36:13 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:13 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:13.259352 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:13.253578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:57239)
0619 12:36:13.253905 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:13.253920 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:13.254019 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:13.254290 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:13.254292 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:13.254481 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:13.254580 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:13.254584 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:13.255310 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:13.255312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:13.255915 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:13.255924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:13.256403 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:13.256407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:13.256497 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:13.257198 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:13.257213 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:13.259180 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":114,"threads_started":1}
W20260619 12:36:13.259491 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:13.253747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57239 (local address 127.0.98.62:37907)
0619 12:36:13.253981 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:13.253985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:13.254000 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:36:13.254126 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:13.254128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:13.254171 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:13.254253 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:13.254686 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:13.255210 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:13.256030 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:13.256255 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:13.256395 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:13.259310 (+ 2915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57239: BlockingRecv error: recv got EOF from 127.0.98.1:57239 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":61,"threads_started":1}
Jun 19 12:36:13 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:13 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:13.468864 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:13.463591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:54789)
0619 12:36:13.463901 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:13.463915 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:13.464016 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:13.464275 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:13.464278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:13.464464 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:13.464555 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:13.464560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:13.465324 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:13.465326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:13.465884 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:13.465892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:13.466354 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:13.466356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:13.466417 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:13.466909 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:13.466921 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:13.468742 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":127,"threads_started":1}
W20260619 12:36:13.469018 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:13.463649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54789 (local address 127.0.98.62:37907)
0619 12:36:13.463832 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:13.463837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:13.463994 (+ 157us) server_negotiation.cc:407] Connection header received
0619 12:36:13.464103 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:13.464105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:13.464167 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:13.464245 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:13.464667 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:13.465212 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:13.466012 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:13.466231 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:13.466343 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:13.468882 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54789: BlockingRecv error: recv got EOF from 127.0.98.2:54789 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 12:36:13 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:13 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:13.678728 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:13.666208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:40007)
0619 12:36:13.666507 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:13.666522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:13.666629 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:13.666895 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:13.666897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:13.667077 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:13.667174 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:13.667179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:13.667959 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:13.667963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:13.668528 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:13.668536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:13.669046 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:13.669050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:13.669119 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:13.676625 (+ 7506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:13.676641 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:13.678604 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":113,"threads_started":1}
W20260619 12:36:13.678933 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:13.666299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40007 (local address 127.0.98.62:37907)
0619 12:36:13.666467 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:13.666471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:13.666601 (+ 130us) server_negotiation.cc:407] Connection header received
0619 12:36:13.666713 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:13.666715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:13.666770 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:13.666856 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:13.667293 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:13.667852 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:13.668670 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:13.668900 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:13.668995 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:13.678788 (+ 9793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40007: BlockingRecv error: recv got EOF from 127.0.98.3:40007 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 12:36:14 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:14 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:14.265923 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:14.260464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:37403)
0619 12:36:14.260749 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:14.260764 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:14.260877 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:14.261137 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:14.261139 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:14.261312 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:14.261408 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:14.261414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:14.262154 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:14.262156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:14.262694 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:14.262701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:14.263123 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:14.263125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:14.263184 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:14.263771 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:14.263782 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:14.265780 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":102,"threads_started":1}
W20260619 12:36:14.266044 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:14.260543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37403 (local address 127.0.98.62:37907)
0619 12:36:14.260816 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:14.260819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:14.260836 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:36:14.260968 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:14.260970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:14.261016 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:14.261099 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:14.261523 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:14.262058 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:14.262799 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:14.263020 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:14.263099 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:14.265914 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37403: BlockingRecv error: recv got EOF from 127.0.98.1:37403 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":110,"threads_started":1}
Jun 19 12:36:14 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:14 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:14.475594 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:14.469835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:45591)
0619 12:36:14.470137 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:14.470155 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:14.470256 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:14.470504 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:14.470506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:14.470687 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:14.470783 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:14.470788 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:14.471672 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:14.471675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:14.472229 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:14.472236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:14.472742 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:14.472745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:14.472817 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:14.473470 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:14.473483 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:14.475399 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":121,"threads_started":1}
W20260619 12:36:14.475840 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:14.470066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45591 (local address 127.0.98.62:37907)
0619 12:36:14.470231 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:14.470234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:14.470251 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:36:14.470331 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:14.470333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:14.470379 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:14.470464 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:14.470945 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:14.471516 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:14.472337 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:14.472576 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:14.472661 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:14.475606 (+ 2945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45591: BlockingRecv error: recv got EOF from 127.0.98.2:45591 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 12:36:14 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:14 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:14.685874 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:14.679815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:50383)
0619 12:36:14.680242 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:14.680259 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:14.680418 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:14.680753 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:14.680757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:14.681016 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:14.681141 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:14.681151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:14.681936 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:14.681939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:14.682501 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:14.682509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:14.682936 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:14.682939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:14.683000 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:14.683640 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:14.683653 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:14.685706 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":107,"threads_started":1}
W20260619 12:36:14.686056 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:14.679871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50383 (local address 127.0.98.62:37907)
0619 12:36:14.680026 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:14.680030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:14.680382 (+ 352us) server_negotiation.cc:407] Connection header received
0619 12:36:14.680511 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:14.680515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:14.680581 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:14.680678 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:14.681280 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:14.681834 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:14.682607 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:14.682841 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:14.682917 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:14.685886 (+ 2969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50383: BlockingRecv error: recv got EOF from 127.0.98.3:50383 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 12:36:15 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:15 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:15.272514 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:15.266905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:39803)
0619 12:36:15.267219 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:15.267233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:15.267337 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:15.267629 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:15.267632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:15.267837 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:15.267938 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:15.267944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:15.268693 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:15.268696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:15.269285 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:15.269293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:15.269702 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:15.269704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:15.269780 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:15.270286 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:15.270298 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:15.272313 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":98,"threads_started":1}
W20260619 12:36:15.272677 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:15.266997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39803 (local address 127.0.98.62:37907)
0619 12:36:15.267285 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:15.267289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:15.267303 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:36:15.267435 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:15.267438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:15.267501 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:15.267581 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:15.268048 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:15.268583 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:15.269390 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:15.269606 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:15.269680 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:15.272510 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39803: BlockingRecv error: recv got EOF from 127.0.98.1:39803 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":62,"threads_started":1}
Jun 19 12:36:15 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:15 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:15.482692 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:15.476561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:53501)
0619 12:36:15.476917 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:15.476931 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:15.477024 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:15.477409 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:15.477415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:15.477630 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:15.477730 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:15.477736 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:15.478814 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:15.478818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:15.479424 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:15.479433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:15.479976 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:15.479978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:15.480043 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:15.480640 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:15.480653 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:15.482547 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":161,"threads_started":1}
W20260619 12:36:15.482843 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:15.476782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53501 (local address 127.0.98.62:37907)
0619 12:36:15.476965 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:15.476969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:15.476989 (+ 20us) server_negotiation.cc:407] Connection header received
0619 12:36:15.477144 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:15.477147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:15.477237 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:15.477325 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:15.477829 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:15.478682 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:15.479621 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:15.479862 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:15.479958 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:15.482716 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53501: BlockingRecv error: recv got EOF from 127.0.98.2:53501 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 12:36:15 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:15 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:15.692276 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:15.686759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.3:46461)
0619 12:36:15.687022 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:15.687039 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:15.687165 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:15.687446 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:15.687449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:15.687656 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:15.687765 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:15.687771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:15.688546 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:15.688549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:15.689128 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:15.689137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:15.689575 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:15.689578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:15.689644 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:15.690148 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:15.690160 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:15.692139 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":107,"threads_started":1}
W20260619 12:36:15.692438 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:15.686807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46461 (local address 127.0.98.62:37907)
0619 12:36:15.686976 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:15.686980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:15.687118 (+ 138us) server_negotiation.cc:407] Connection header received
0619 12:36:15.687235 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:15.687237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:15.687284 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:15.687373 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:15.687883 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:15.688416 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:15.689240 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:15.689463 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:15.689541 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:15.692287 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46461: BlockingRecv error: recv got EOF from 127.0.98.3:46461 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 12:36:16 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:16 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:16.279098 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:16.273491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.1:41893)
0619 12:36:16.273799 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:16.273813 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:16.273903 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:16.274159 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:16.274161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:16.274338 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:16.274442 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:16.274447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:16.275200 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:16.275203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:16.275848 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:16.275858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:16.276301 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:16.276304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:16.276373 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:16.276916 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:16.276928 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:16.278915 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":98,"threads_started":1}
W20260619 12:36:16.279176 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:16.273597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41893 (local address 127.0.98.62:37907)
0619 12:36:16.273825 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:16.273829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:16.273865 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:36:16.273987 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:16.273989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:16.274032 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:16.274113 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:16.274547 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:16.275084 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:16.275953 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:16.276175 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:16.276255 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:16.279051 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41893: BlockingRecv error: recv got EOF from 127.0.98.1:41893 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
Jun 19 12:36:16 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:36:16 dist-test-slave-kdrb krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:36:16.490163 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:16.483793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37907 (local address 127.0.98.2:36585)
0619 12:36:16.484134 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:16.484151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:16.484304 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:16.484580 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:16.484583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:16.484801 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:16.484910 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:16.484957 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:16.485842 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:16.485845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:16.486401 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:16.486409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:16.486929 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:16.486933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:16.487036 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:16.487703 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:16.487717 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:16.490018 (+ 2301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37907: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":86,"threads_started":1}
W20260619 12:36:16.490423 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:16.483931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36585 (local address 127.0.98.62:37907)
0619 12:36:16.484119 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:16.484125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:16.484245 (+ 120us) server_negotiation.cc:407] Connection header received
0619 12:36:16.484418 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:16.484428 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:16.484472 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:16.484563 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:16.485108 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:16.485675 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:16.486504 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:16.486792 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:16.486934 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:16.490284 (+ 3350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36585: BlockingRecv error: recv got EOF from 127.0.98.2:36585 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 12:36:16.570098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 9141
I20260619 12:36:16.576294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 9293
I20260619 12:36:16.582108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 9425
I20260619 12:36:16.587765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 9071
2026-06-19T12:36:16Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18070 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T12:36:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:36:16Z Disabled control of system clock
I20260619 12:36:16.618868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34303
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 12:36:16.754750 9621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:16.755118 9621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:16.755211 9621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:16.759730 9621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:36:16.759826 9621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:16.759855 9621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:16.759907 9621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:36:16.759948 9621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:36:16.765378 9621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:16.766566 9621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:16.768087 9621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:36:16.777189 9621 server_base.cc:1034] running on GCE node
W20260619 12:36:16.777427 9627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:16.777204 9626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:16.777325 9629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:16.778205 9621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:16.779395 9621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:16.780589 9621 hybrid_clock.cc:648] HybridClock initialized: now 1781872576780567 us; error 55 us; skew 500 ppm
I20260619 12:36:16.782640 9621 webserver.cc:460] Webserver started at http://127.0.98.62:46289/ using document root <none> and password file <none>
I20260619 12:36:16.783267 9621 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:16.783370 9621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:16.783663 9621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:36:16.785915 9621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "1bc549a6cfca43e4a053c973353b11f8"
format_stamp: "Formatted at 2026-06-19 12:36:16 on dist-test-slave-kdrb"
server_key: "a0aab058a4830097be340bf349e9e1a7"
server_key_iv: "0a96a7bb0c307e3ffcdcfcacc1ffe4fe"
server_key_version: "encryptionkey@0"
I20260619 12:36:16.786509 9621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "1bc549a6cfca43e4a053c973353b11f8"
format_stamp: "Formatted at 2026-06-19 12:36:16 on dist-test-slave-kdrb"
server_key: "a0aab058a4830097be340bf349e9e1a7"
server_key_iv: "0a96a7bb0c307e3ffcdcfcacc1ffe4fe"
server_key_version: "encryptionkey@0"
I20260619 12:36:16.790570 9621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 12:36:16.793280 9635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:16.794727 9621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 12:36:16.794881 9621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "1bc549a6cfca43e4a053c973353b11f8"
format_stamp: "Formatted at 2026-06-19 12:36:16 on dist-test-slave-kdrb"
server_key: "a0aab058a4830097be340bf349e9e1a7"
server_key_iv: "0a96a7bb0c307e3ffcdcfcacc1ffe4fe"
server_key_version: "encryptionkey@0"
I20260619 12:36:16.795003 9621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:16.814862 9621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:16.815732 9621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:16.815941 9621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:16.823065 9621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34303
I20260619 12:36:16.823071 9687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34303 every 8 connection(s)
I20260619 12:36:16.824110 9621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:36:16.825253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 9621
I20260619 12:36:16.825357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:36:16.825652 392 external_mini_cluster.cc:1428] Setting key 8a809a728ea92abd941e21d963c3cb8d
I20260619 12:36:16.827561 9688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:36:16.833353 9688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8: Bootstrap starting.
I20260619 12:36:16.835007 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:16.826955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48370 (local address 127.0.98.62:34303)
0619 12:36:16.827641 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:16.827661 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:16.827711 (+ 50us) server_negotiation.cc:407] Connection header received
0619 12:36:16.828393 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:16.828410 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:16.828812 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:16.829160 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:16.829588 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:16.830565 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:16.831524 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:16.832022 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:16.832210 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:16.833154 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:16.833160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:16.833173 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:36:16.833645 (+ 472us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:16.833706 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:36:16.833727 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:16.833841 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:16.834003 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 12:36:16.834272 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":125,"threads_started":1}
I20260619 12:36:16.835871 9688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8: Neither blocks nor log segments found. Creating new log.
I20260619 12:36:16.836663 9688 log.cc:824] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8: Log is configured to *not* fsync() on all Append() calls
I20260619 12:36:16.838688 9688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8: No bootstrap required, opened a new log
I20260619 12:36:16.842350 9688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc549a6cfca43e4a053c973353b11f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34303 } }
I20260619 12:36:16.842605 9688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:36:16.842685 9688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc549a6cfca43e4a053c973353b11f8, State: Initialized, Role: FOLLOWER
I20260619 12:36:16.843099 9688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc549a6cfca43e4a053c973353b11f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34303 } }
I20260619 12:36:16.843252 9688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:36:16.843333 9688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:36:16.843451 9688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:36:16.844385 9688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc549a6cfca43e4a053c973353b11f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34303 } }
I20260619 12:36:16.844713 9688 leader_election.cc:302] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bc549a6cfca43e4a053c973353b11f8; no voters:
I20260619 12:36:16.845058 9688 leader_election.cc:288] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:36:16.845173 9693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:36:16.845492 9693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [term 1 LEADER]: Becoming Leader. State: Replica: 1bc549a6cfca43e4a053c973353b11f8, State: Running, Role: LEADER
I20260619 12:36:16.845847 9693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc549a6cfca43e4a053c973353b11f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34303 } }
I20260619 12:36:16.846397 9688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:36:16.847493 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bc549a6cfca43e4a053c973353b11f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc549a6cfca43e4a053c973353b11f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34303 } } }
I20260619 12:36:16.847635 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [sys.catalog]: This master's current role is: LEADER
I20260619 12:36:16.847954 9695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bc549a6cfca43e4a053c973353b11f8. Latest consensus state: current_term: 1 leader_uuid: "1bc549a6cfca43e4a053c973353b11f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc549a6cfca43e4a053c973353b11f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34303 } } }
I20260619 12:36:16.848085 9695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8 [sys.catalog]: This master's current role is: LEADER
I20260619 12:36:16.848057 9702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:36:16.851424 9702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:36:16.857656 9702 catalog_manager.cc:1347] Generated new cluster ID: 3b8f547900bf46f5a9943e5f355d7f84
I20260619 12:36:16.857836 9702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:36:16.877079 9702 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:36:16.878082 9702 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:36:16.887059 9702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bc549a6cfca43e4a053c973353b11f8: Generated new TSK 0
I20260619 12:36:16.887941 9702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:36:16.900441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34303
--builtin_ntp_servers=127.0.98.20:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 12:36:17.015864 9712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:17.016144 9712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:17.016182 9712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:17.020310 9712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:17.020378 9712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:17.020493 9712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:36:17.025341 9712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:17.026492 9712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:17.027944 9712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:17.037755 9718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:17.037722 9717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:17.037899 9720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:17.037822 9712 server_base.cc:1034] running on GCE node
I20260619 12:36:17.038458 9712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:17.039132 9712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:17.040328 9712 hybrid_clock.cc:648] HybridClock initialized: now 1781872577040282 us; error 56 us; skew 500 ppm
I20260619 12:36:17.042330 9712 webserver.cc:460] Webserver started at http://127.0.98.1:41305/ using document root <none> and password file <none>
I20260619 12:36:17.042929 9712 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:17.043026 9712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:17.043259 9712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:36:17.045411 9712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "4f9f0331447047deaf54ba8a3320ea5c"
format_stamp: "Formatted at 2026-06-19 12:36:17 on dist-test-slave-kdrb"
server_key: "8b426c081e50bb29940351d2a9accdc9"
server_key_iv: "8a817cdba73f532d938a948927bd6240"
server_key_version: "encryptionkey@0"
I20260619 12:36:17.045995 9712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4f9f0331447047deaf54ba8a3320ea5c"
format_stamp: "Formatted at 2026-06-19 12:36:17 on dist-test-slave-kdrb"
server_key: "8b426c081e50bb29940351d2a9accdc9"
server_key_iv: "8a817cdba73f532d938a948927bd6240"
server_key_version: "encryptionkey@0"
I20260619 12:36:17.049957 9712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:36:17.052745 9726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:17.054327 9712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 12:36:17.054461 9712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "4f9f0331447047deaf54ba8a3320ea5c"
format_stamp: "Formatted at 2026-06-19 12:36:17 on dist-test-slave-kdrb"
server_key: "8b426c081e50bb29940351d2a9accdc9"
server_key_iv: "8a817cdba73f532d938a948927bd6240"
server_key_version: "encryptionkey@0"
I20260619 12:36:17.054589 9712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:17.068192 9712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:17.068892 9712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:17.069082 9712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:17.069669 9712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:36:17.070602 9712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:36:17.070688 9712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:17.070758 9712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:36:17.070791 9712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:17.079679 9712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38421
I20260619 12:36:17.079696 9839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38421 every 8 connection(s)
I20260619 12:36:17.080648 9712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:36:17.087355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 9712
I20260619 12:36:17.087515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:36:17.087842 392 external_mini_cluster.cc:1428] Setting key a1684622347a9103be297bf88386e7e3
I20260619 12:36:17.089972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34303
--builtin_ntp_servers=127.0.98.20:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 12:36:17.090802 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:17.082556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47079 (local address 127.0.98.62:34303)
0619 12:36:17.082715 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:17.082720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:17.083990 (+ 1270us) server_negotiation.cc:407] Connection header received
0619 12:36:17.085422 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:17.085425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:17.085479 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:17.085563 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:17.086541 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:17.087066 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:17.088708 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:17.088940 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:17.089059 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:17.090310 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:17.090313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:17.090315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:36:17.090328 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:17.090347 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:36:17.090356 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:17.090417 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:17.090604 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 12:36:17.090676 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 12:36:17.091985 9842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:17.083019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34303 (local address 127.0.98.1:47079)
0619 12:36:17.083821 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:17.083867 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:17.085201 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:17.085782 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:17.085789 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:17.085886 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:17.086411 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:17.086424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:17.087220 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:17.087224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:17.088562 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:17.088573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:17.089099 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:17.089103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:17.089309 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:17.089984 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:36:17.089989 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:17.090051 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:36:17.090149 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:36:17.090444 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:36:17.090455 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:36:17.090464 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 12:36:17.090602 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 12:36:17.090938 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":113,"threads_started":1}
I20260619 12:36:17.093426 9840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34303
I20260619 12:36:17.093701 9840 heartbeater.cc:464] Registering TS with master...
I20260619 12:36:17.094430 9840 heartbeater.cc:511] Master 127.0.98.62:34303 requested a full tablet report, sending...
I20260619 12:36:17.096341 9651 ts_manager.cc:194] Registered new tserver with Master: 4f9f0331447047deaf54ba8a3320ea5c (127.0.98.1:38421)
I20260619 12:36:17.097523 9651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47079
W20260619 12:36:17.204033 9843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:17.204322 9843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:17.204361 9843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:17.208485 9843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:17.208545 9843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:17.208622 9843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:36:17.213523 9843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:17.214635 9843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:17.216131 9843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:17.226051 9849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:17.226073 9851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:17.226131 9843 server_base.cc:1034] running on GCE node
W20260619 12:36:17.226238 9848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:17.226718 9843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:17.227341 9843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:17.228502 9843 hybrid_clock.cc:648] HybridClock initialized: now 1781872577228485 us; error 31 us; skew 500 ppm
I20260619 12:36:17.230263 9843 webserver.cc:460] Webserver started at http://127.0.98.2:34625/ using document root <none> and password file <none>
I20260619 12:36:17.230806 9843 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:17.230870 9843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:17.231034 9843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:36:17.233199 9843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "c8ff496b0d9d4c35af54bfc37ba1373c"
format_stamp: "Formatted at 2026-06-19 12:36:17 on dist-test-slave-kdrb"
server_key: "db78b76dfff4e74846af2ec6fed77150"
server_key_iv: "0afe4f24dffcbc9cbd21159336f6e65d"
server_key_version: "encryptionkey@0"
I20260619 12:36:17.233837 9843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c8ff496b0d9d4c35af54bfc37ba1373c"
format_stamp: "Formatted at 2026-06-19 12:36:17 on dist-test-slave-kdrb"
server_key: "db78b76dfff4e74846af2ec6fed77150"
server_key_iv: "0afe4f24dffcbc9cbd21159336f6e65d"
server_key_version: "encryptionkey@0"
I20260619 12:36:17.237679 9843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:36:17.240216 9857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:17.241449 9843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:36:17.241545 9843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "c8ff496b0d9d4c35af54bfc37ba1373c"
format_stamp: "Formatted at 2026-06-19 12:36:17 on dist-test-slave-kdrb"
server_key: "db78b76dfff4e74846af2ec6fed77150"
server_key_iv: "0afe4f24dffcbc9cbd21159336f6e65d"
server_key_version: "encryptionkey@0"
I20260619 12:36:17.241639 9843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:17.252583 9843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:17.253242 9843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:17.253404 9843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:17.253925 9843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:36:17.254812 9843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:36:17.254858 9843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:17.254897 9843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:36:17.254912 9843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:17.263945 9843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40281
I20260619 12:36:17.264007 9970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40281 every 8 connection(s)
I20260619 12:36:17.264911 9843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:36:17.266845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 9843
I20260619 12:36:17.266957 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:36:17.267271 392 external_mini_cluster.cc:1428] Setting key f1529d47d5decd626c8504ecd4fd5b7a
I20260619 12:36:17.269384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34303
--builtin_ntp_servers=127.0.98.20:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 12:36:17.274888 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:17.266763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45409 (local address 127.0.98.62:34303)
0619 12:36:17.266922 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:17.266929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:17.268365 (+ 1436us) server_negotiation.cc:407] Connection header received
0619 12:36:17.269963 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:17.269970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:17.270033 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:17.270126 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:17.271043 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:17.271643 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:17.272831 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:17.273058 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:17.273173 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:17.274392 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:17.274401 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:17.274403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:36:17.274416 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:17.274434 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:36:17.274442 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:17.274519 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:17.274685 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 12:36:17.274746 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 12:36:17.275923 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:17.267309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34303 (local address 127.0.98.2:45409)
0619 12:36:17.268195 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:17.268252 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:17.269744 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:17.270284 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:17.270291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:17.270377 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:17.270897 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:17.270910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:17.271770 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:17.271774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:17.272697 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:17.272706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:17.273212 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:17.273216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:17.273397 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:17.274085 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:36:17.274086 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:17.274128 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:36:17.274228 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:36:17.274532 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:36:17.274543 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:36:17.274549 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:36:17.274685 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 12:36:17.275073 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":166,"threads_started":1}
I20260619 12:36:17.277112 9971 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34303
I20260619 12:36:17.277393 9971 heartbeater.cc:464] Registering TS with master...
I20260619 12:36:17.278071 9971 heartbeater.cc:511] Master 127.0.98.62:34303 requested a full tablet report, sending...
I20260619 12:36:17.279233 9651 ts_manager.cc:194] Registered new tserver with Master: c8ff496b0d9d4c35af54bfc37ba1373c (127.0.98.2:40281)
I20260619 12:36:17.279866 9651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45409
W20260619 12:36:17.388090 9974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:17.388381 9974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:17.388458 9974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:17.392629 9974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:17.392697 9974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:17.392781 9974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:36:17.397863 9974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:17.399137 9974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:17.400630 9974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:17.410507 9980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:17.410544 9982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:17.410535 9974 server_base.cc:1034] running on GCE node
W20260619 12:36:17.410633 9979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:17.411155 9974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:17.411880 9974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:17.413065 9974 hybrid_clock.cc:648] HybridClock initialized: now 1781872577413043 us; error 40 us; skew 500 ppm
I20260619 12:36:17.414985 9974 webserver.cc:460] Webserver started at http://127.0.98.3:41667/ using document root <none> and password file <none>
I20260619 12:36:17.415643 9974 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:17.415737 9974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:17.415974 9974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:36:17.418061 9974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "9b698cc6336c4d31a36884cf5bd91121"
format_stamp: "Formatted at 2026-06-19 12:36:17 on dist-test-slave-kdrb"
server_key: "0768d321a0cd45c01ba7fe3728d0fff6"
server_key_iv: "40310ea528eda02bb7a6ebfbcabedc0b"
server_key_version: "encryptionkey@0"
I20260619 12:36:17.418635 9974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9b698cc6336c4d31a36884cf5bd91121"
format_stamp: "Formatted at 2026-06-19 12:36:17 on dist-test-slave-kdrb"
server_key: "0768d321a0cd45c01ba7fe3728d0fff6"
server_key_iv: "40310ea528eda02bb7a6ebfbcabedc0b"
server_key_version: "encryptionkey@0"
I20260619 12:36:17.422587 9974 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:36:17.425554 9988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:17.427066 9974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:36:17.427217 9974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "9b698cc6336c4d31a36884cf5bd91121"
format_stamp: "Formatted at 2026-06-19 12:36:17 on dist-test-slave-kdrb"
server_key: "0768d321a0cd45c01ba7fe3728d0fff6"
server_key_iv: "40310ea528eda02bb7a6ebfbcabedc0b"
server_key_version: "encryptionkey@0"
I20260619 12:36:17.427361 9974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:17.438181 9974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:17.438966 9974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:17.439177 9974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:17.439872 9974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:36:17.440871 9974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:36:17.440917 9974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:17.440989 9974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:36:17.441026 9974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:17.450105 9974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45637
I20260619 12:36:17.450163 10101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45637 every 8 connection(s)
I20260619 12:36:17.451102 9974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:36:17.456072 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 9974
I20260619 12:36:17.456197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:36:17.456496 392 external_mini_cluster.cc:1428] Setting key 2d42f90b8ae76fea318dd41d02fad5dc
I20260619 12:36:17.461145 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:17.453023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36943 (local address 127.0.98.62:34303)
0619 12:36:17.453181 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:17.453186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:17.454359 (+ 1173us) server_negotiation.cc:407] Connection header received
0619 12:36:17.455764 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:17.455766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:17.455821 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:17.455900 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:17.457122 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:17.457885 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:17.459319 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:17.459612 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:17.459695 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:17.460703 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:17.460706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:17.460708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:36:17.460720 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:17.460737 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:36:17.460745 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:17.460812 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:17.460980 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 12:36:17.461033 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 12:36:17.461963 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:17.453522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34303 (local address 127.0.98.3:36943)
0619 12:36:17.454193 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:17.454234 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:17.455567 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:17.456119 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:17.456130 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:17.456253 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:17.456949 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:17.456969 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:17.458002 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:17.458005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:17.459148 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:17.459157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:17.459746 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:17.459749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:17.459901 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:17.460460 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:36:17.460461 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:17.460502 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:36:17.460566 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:36:17.460822 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:36:17.460828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:36:17.460831 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:36:17.460927 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 12:36:17.461151 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
I20260619 12:36:17.463069 10102 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34303
I20260619 12:36:17.463305 10102 heartbeater.cc:464] Registering TS with master...
I20260619 12:36:17.463887 10102 heartbeater.cc:511] Master 127.0.98.62:34303 requested a full tablet report, sending...
I20260619 12:36:17.464890 9651 ts_manager.cc:194] Registered new tserver with Master: 9b698cc6336c4d31a36884cf5bd91121 (127.0.98.3:45637)
I20260619 12:36:17.465430 9651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36943
I20260619 12:36:17.470227 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 12:36:17.473089 9691 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:17.472135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.0.98.62:34303)
0619 12:36:17.472279 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:17.472284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:17.472459 (+ 175us) server_negotiation.cc:407] Connection header received
0619 12:36:17.472550 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:17.472553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:17.472594 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:17.472660 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:17.472930 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48382: BlockingRecv error: recv got EOF from 127.0.0.1:48382 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 12:36:17.473229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 9712
I20260619 12:36:17.478830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 9843
I20260619 12:36:17.484220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 9974
I20260619 12:36:17.489521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 9621
2026-06-19T12:36:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (900 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T12:36:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:36:17Z Disabled control of system clock
I20260619 12:36:17.518986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33017
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 12:36:17.630846 10113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:17.631117 10113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:17.631155 10113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:17.635301 10113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:36:17.635368 10113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:17.635385 10113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:17.635401 10113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:36:17.635417 10113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:36:17.640198 10113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33017
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:17.641335 10113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:17.642735 10113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:17.651363 10121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:17.651378 10118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:17.651381 10119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:17.651659 10113 server_base.cc:1034] running on GCE node
I20260619 12:36:17.652230 10113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:17.653430 10113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:17.654644 10113 hybrid_clock.cc:648] HybridClock initialized: now 1781872577654633 us; error 32 us; skew 500 ppm
I20260619 12:36:17.656749 10113 webserver.cc:460] Webserver started at http://127.0.98.62:40795/ using document root <none> and password file <none>
I20260619 12:36:17.657379 10113 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:17.657471 10113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:17.657688 10113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:36:17.659855 10113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "e323e6a47871404687d786e571030b35"
format_stamp: "Formatted at 2026-06-19 12:36:17 on dist-test-slave-kdrb"
server_key: "d33d52aa833310e6981d501f8a9675e6"
server_key_iv: "28ebab590f2ad82602b0ab8f6e28b332"
server_key_version: "encryptionkey@0"
I20260619 12:36:17.660429 10113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "e323e6a47871404687d786e571030b35"
format_stamp: "Formatted at 2026-06-19 12:36:17 on dist-test-slave-kdrb"
server_key: "d33d52aa833310e6981d501f8a9675e6"
server_key_iv: "28ebab590f2ad82602b0ab8f6e28b332"
server_key_version: "encryptionkey@0"
I20260619 12:36:17.664258 10113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:36:17.666780 10127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:17.668108 10113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:36:17.668268 10113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "e323e6a47871404687d786e571030b35"
format_stamp: "Formatted at 2026-06-19 12:36:17 on dist-test-slave-kdrb"
server_key: "d33d52aa833310e6981d501f8a9675e6"
server_key_iv: "28ebab590f2ad82602b0ab8f6e28b332"
server_key_version: "encryptionkey@0"
I20260619 12:36:17.668406 10113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:17.673182 10113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:17.673834 10113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:17.674018 10113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:17.680943 10113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33017
I20260619 12:36:17.680949 10179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33017 every 8 connection(s)
I20260619 12:36:17.681952 10113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:36:17.685060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 10113
I20260619 12:36:17.685207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:36:17.685415 10180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:36:17.685559 392 external_mini_cluster.cc:1428] Setting key f9177880a9193accb2377a35a0bc5fcc
I20260619 12:36:17.691254 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:17.686980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.0.98.62:33017)
0619 12:36:17.687732 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:17.687751 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:17.687813 (+ 62us) server_negotiation.cc:407] Connection header received
0619 12:36:17.688451 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:17.688469 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:17.688840 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:17.689185 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:17.689980 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:17.689985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:17.689998 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:36:17.690008 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:17.690061 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:36:17.690077 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:17.690232 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:17.690303 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 12:36:17.690548 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":132,"threads_started":1}
I20260619 12:36:17.692374 10180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35: Bootstrap starting.
I20260619 12:36:17.694777 10180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35: Neither blocks nor log segments found. Creating new log.
I20260619 12:36:17.695739 10180 log.cc:824] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35: Log is configured to *not* fsync() on all Append() calls
I20260619 12:36:17.697710 10180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35: No bootstrap required, opened a new log
I20260619 12:36:17.701185 10180 raft_consensus.cc:348] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e323e6a47871404687d786e571030b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33017 } }
I20260619 12:36:17.701421 10180 raft_consensus.cc:374] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:36:17.701457 10180 raft_consensus.cc:729] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e323e6a47871404687d786e571030b35, State: Initialized, Role: FOLLOWER
I20260619 12:36:17.701857 10180 consensus_queue.cc:260] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e323e6a47871404687d786e571030b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33017 } }
I20260619 12:36:17.702013 10180 raft_consensus.cc:388] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:36:17.702082 10180 raft_consensus.cc:482] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:36:17.702204 10180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:36:17.703269 10180 raft_consensus.cc:504] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e323e6a47871404687d786e571030b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33017 } }
I20260619 12:36:17.703637 10180 leader_election.cc:302] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e323e6a47871404687d786e571030b35; no voters:
I20260619 12:36:17.703912 10180 leader_election.cc:288] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:36:17.704051 10185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:36:17.704305 10185 raft_consensus.cc:686] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [term 1 LEADER]: Becoming Leader. State: Replica: e323e6a47871404687d786e571030b35, State: Running, Role: LEADER
I20260619 12:36:17.704654 10185 consensus_queue.cc:237] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e323e6a47871404687d786e571030b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33017 } }
I20260619 12:36:17.705138 10180 sys_catalog.cc:564] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:36:17.706256 10187 sys_catalog.cc:455] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e323e6a47871404687d786e571030b35. Latest consensus state: current_term: 1 leader_uuid: "e323e6a47871404687d786e571030b35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e323e6a47871404687d786e571030b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33017 } } }
I20260619 12:36:17.706401 10187 sys_catalog.cc:458] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [sys.catalog]: This master's current role is: LEADER
I20260619 12:36:17.706626 10186 sys_catalog.cc:455] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e323e6a47871404687d786e571030b35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e323e6a47871404687d786e571030b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33017 } } }
I20260619 12:36:17.706774 10186 sys_catalog.cc:458] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35 [sys.catalog]: This master's current role is: LEADER
I20260619 12:36:17.706837 10194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:36:17.710116 10194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:36:17.716130 10194 catalog_manager.cc:1347] Generated new cluster ID: ece6c4446c124176b7fdb3448b5de64c
I20260619 12:36:17.716219 10194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:36:17.730217 10194 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:36:17.730580 10194 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:36:17.738121 10194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e323e6a47871404687d786e571030b35: Generated new TSK 0
I20260619 12:36:17.738835 10194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:36:17.747064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33017
--builtin_ntp_servers=127.0.98.20:46809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 12:36:17.864692 10204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:17.864967 10204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:17.865008 10204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:17.869086 10204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:17.869148 10204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:17.869203 10204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:36:17.874094 10204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:17.875129 10204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:17.876525 10204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:17.886581 10209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:17.886603 10212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:17.886634 10204 server_base.cc:1034] running on GCE node
W20260619 12:36:17.886595 10210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:17.887275 10204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:17.888026 10204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:17.889220 10204 hybrid_clock.cc:648] HybridClock initialized: now 1781872577889202 us; error 32 us; skew 500 ppm
I20260619 12:36:17.891198 10204 webserver.cc:460] Webserver started at http://127.0.98.1:44065/ using document root <none> and password file <none>
I20260619 12:36:17.891901 10204 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:17.891996 10204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:17.892234 10204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:36:17.894475 10204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "4df9378cac7a420bba7ea8fb4c8db6f5"
format_stamp: "Formatted at 2026-06-19 12:36:17 on dist-test-slave-kdrb"
server_key: "79a99a9a1c393ce307be3ae26cf1ee25"
server_key_iv: "62842dd40f7cd7a79ec6364409c2fd33"
server_key_version: "encryptionkey@0"
I20260619 12:36:17.895067 10204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4df9378cac7a420bba7ea8fb4c8db6f5"
format_stamp: "Formatted at 2026-06-19 12:36:17 on dist-test-slave-kdrb"
server_key: "79a99a9a1c393ce307be3ae26cf1ee25"
server_key_iv: "62842dd40f7cd7a79ec6364409c2fd33"
server_key_version: "encryptionkey@0"
I20260619 12:36:17.899068 10204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 12:36:17.901507 10218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:17.902740 10204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:36:17.902885 10204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "4df9378cac7a420bba7ea8fb4c8db6f5"
format_stamp: "Formatted at 2026-06-19 12:36:17 on dist-test-slave-kdrb"
server_key: "79a99a9a1c393ce307be3ae26cf1ee25"
server_key_iv: "62842dd40f7cd7a79ec6364409c2fd33"
server_key_version: "encryptionkey@0"
I20260619 12:36:17.903004 10204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:17.907665 10204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:17.908255 10204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:17.908449 10204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:17.908993 10204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:36:17.909914 10204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:36:17.909984 10204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:17.910050 10204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:36:17.910099 10204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:17.918658 10204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36191
I20260619 12:36:17.918678 10331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36191 every 8 connection(s)
I20260619 12:36:17.919667 10204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:36:17.923807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 10204
I20260619 12:36:17.923935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:36:17.924322 392 external_mini_cluster.cc:1428] Setting key 5383b0b0361316c92d9410c846dbc40f
I20260619 12:36:17.926187 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:17.921495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57119 (local address 127.0.98.62:33017)
0619 12:36:17.921648 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:17.921652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:17.922724 (+ 1072us) server_negotiation.cc:407] Connection header received
0619 12:36:17.924288 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:17.924291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:17.924338 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:17.924439 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:17.925670 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:17.925674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:17.925677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:36:17.925692 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:17.925715 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:36:17.925726 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:17.925818 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:17.926014 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 12:36:17.926058 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 12:36:17.926578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33017
--builtin_ntp_servers=127.0.98.20:46809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 12:36:17.927114 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:17.921916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33017 (local address 127.0.98.1:57119)
0619 12:36:17.922568 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:17.922608 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:17.924032 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:17.924618 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:17.924626 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:17.924721 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:17.925363 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:36:17.925364 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:17.925408 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:36:17.925480 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:36:17.925836 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:36:17.925845 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:36:17.925851 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:36:17.926036 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 12:36:17.926254 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":112,"threads_started":1}
I20260619 12:36:17.928514 10332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33017
I20260619 12:36:17.928772 10332 heartbeater.cc:464] Registering TS with master...
I20260619 12:36:17.929180 10332 heartbeater.cc:511] Master 127.0.98.62:33017 requested a full tablet report, sending...
I20260619 12:36:17.930894 10144 ts_manager.cc:194] Registered new tserver with Master: 4df9378cac7a420bba7ea8fb4c8db6f5 (127.0.98.1:36191)
W20260619 12:36:18.039036 10335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:18.039345 10335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:18.039386 10335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:18.043599 10335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:18.043674 10335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:18.043735 10335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:36:18.048730 10335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:18.049882 10335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:18.051287 10335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:18.061369 10343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:18.061380 10341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:18.061380 10340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:18.061444 10335 server_base.cc:1034] running on GCE node
I20260619 12:36:18.062109 10335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:18.062839 10335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:18.064030 10335 hybrid_clock.cc:648] HybridClock initialized: now 1781872578064013 us; error 33 us; skew 500 ppm
I20260619 12:36:18.065981 10335 webserver.cc:460] Webserver started at http://127.0.98.2:43999/ using document root <none> and password file <none>
I20260619 12:36:18.066617 10335 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:18.066717 10335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:18.066962 10335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:36:18.069155 10335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "fa749eca5cc94adf8ce1b5f423ce844c"
format_stamp: "Formatted at 2026-06-19 12:36:18 on dist-test-slave-kdrb"
server_key: "276ba0b5d15abcf97fd6394852f2f00a"
server_key_iv: "aec5d208f73c569c233be6b6e017b98d"
server_key_version: "encryptionkey@0"
I20260619 12:36:18.069753 10335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fa749eca5cc94adf8ce1b5f423ce844c"
format_stamp: "Formatted at 2026-06-19 12:36:18 on dist-test-slave-kdrb"
server_key: "276ba0b5d15abcf97fd6394852f2f00a"
server_key_iv: "aec5d208f73c569c233be6b6e017b98d"
server_key_version: "encryptionkey@0"
I20260619 12:36:18.073805 10335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:36:18.076279 10349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:18.077541 10335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:36:18.077647 10335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "fa749eca5cc94adf8ce1b5f423ce844c"
format_stamp: "Formatted at 2026-06-19 12:36:18 on dist-test-slave-kdrb"
server_key: "276ba0b5d15abcf97fd6394852f2f00a"
server_key_iv: "aec5d208f73c569c233be6b6e017b98d"
server_key_version: "encryptionkey@0"
I20260619 12:36:18.077744 10335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:18.082327 10335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:18.082949 10335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:18.083127 10335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:18.083750 10335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:36:18.084666 10335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:36:18.084712 10335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:18.084785 10335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:36:18.084821 10335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:18.093775 10335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35403
I20260619 12:36:18.093791 10462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35403 every 8 connection(s)
I20260619 12:36:18.094772 10335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:36:18.100905 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:18.096629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39951 (local address 127.0.98.62:33017)
0619 12:36:18.096779 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:18.096783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:18.097938 (+ 1155us) server_negotiation.cc:407] Connection header received
0619 12:36:18.099306 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:18.099309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:18.099350 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:18.099428 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:18.100530 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:18.100533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:18.100535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:36:18.100545 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:18.100565 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:36:18.100573 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:18.100650 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:18.100738 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 12:36:18.100775 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:36:18.101765 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:18.097113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33017 (local address 127.0.98.2:39951)
0619 12:36:18.097771 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:18.097830 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:18.099104 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:18.099644 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:18.099652 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:18.099744 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:18.100278 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:36:18.100280 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:18.100323 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:36:18.100393 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:36:18.100632 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:36:18.100636 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:36:18.100639 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:36:18.100760 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 12:36:18.100976 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":117,"threads_started":1}
I20260619 12:36:18.102890 10463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33017
I20260619 12:36:18.103193 10463 heartbeater.cc:464] Registering TS with master...
I20260619 12:36:18.103358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 10335
I20260619 12:36:18.103495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:36:18.103727 10463 heartbeater.cc:511] Master 127.0.98.62:33017 requested a full tablet report, sending...
I20260619 12:36:18.103808 392 external_mini_cluster.cc:1428] Setting key 0d418a9ffb7096d355fc136278d8da20
I20260619 12:36:18.104794 10143 ts_manager.cc:194] Registered new tserver with Master: fa749eca5cc94adf8ce1b5f423ce844c (127.0.98.2:35403)
I20260619 12:36:18.105952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33017
--builtin_ntp_servers=127.0.98.20:46809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 12:36:18.217826 10466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:18.218137 10466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:18.218220 10466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:18.222766 10466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:18.222842 10466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:18.222937 10466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:36:18.227752 10466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:18.228879 10466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:18.230261 10466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:18.239996 10474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:18.240051 10466 server_base.cc:1034] running on GCE node
W20260619 12:36:18.239996 10472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:18.240159 10471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:18.240777 10466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:18.241483 10466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:18.242671 10466 hybrid_clock.cc:648] HybridClock initialized: now 1781872578242634 us; error 56 us; skew 500 ppm
I20260619 12:36:18.244647 10466 webserver.cc:460] Webserver started at http://127.0.98.3:43777/ using document root <none> and password file <none>
I20260619 12:36:18.245270 10466 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:18.245379 10466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:18.245632 10466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:36:18.247903 10466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "29327d85da16443a95c14d751bf63769"
format_stamp: "Formatted at 2026-06-19 12:36:18 on dist-test-slave-kdrb"
server_key: "c46fdfe06fe19b76f26780b5d4b77913"
server_key_iv: "1f4ed3c96f467937d93a34814576cdcb"
server_key_version: "encryptionkey@0"
I20260619 12:36:18.248520 10466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "29327d85da16443a95c14d751bf63769"
format_stamp: "Formatted at 2026-06-19 12:36:18 on dist-test-slave-kdrb"
server_key: "c46fdfe06fe19b76f26780b5d4b77913"
server_key_iv: "1f4ed3c96f467937d93a34814576cdcb"
server_key_version: "encryptionkey@0"
I20260619 12:36:18.252681 10466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 12:36:18.255364 10480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:18.256907 10466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:36:18.257085 10466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "29327d85da16443a95c14d751bf63769"
format_stamp: "Formatted at 2026-06-19 12:36:18 on dist-test-slave-kdrb"
server_key: "c46fdfe06fe19b76f26780b5d4b77913"
server_key_iv: "1f4ed3c96f467937d93a34814576cdcb"
server_key_version: "encryptionkey@0"
I20260619 12:36:18.257263 10466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:18.262301 10466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:18.262981 10466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:18.263188 10466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:18.263842 10466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:36:18.264791 10466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:36:18.264863 10466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:18.264940 10466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:36:18.264982 10466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:18.273670 10466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39469
I20260619 12:36:18.273718 10593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39469 every 8 connection(s)
I20260619 12:36:18.274633 10466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:36:18.280885 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:18.276492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60667 (local address 127.0.98.62:33017)
0619 12:36:18.276634 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:18.276637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:18.277658 (+ 1021us) server_negotiation.cc:407] Connection header received
0619 12:36:18.278903 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:18.278905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:18.278942 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:18.279017 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:18.280307 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:18.280310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:18.280312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:36:18.280326 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:18.280347 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:36:18.280355 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:18.280438 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:18.280714 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 12:36:18.280761 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:36:18.281921 10596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:18.276856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33017 (local address 127.0.98.3:60667)
0619 12:36:18.277509 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:18.277550 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:18.278716 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:18.279225 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:18.279232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:18.279326 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:18.280032 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:36:18.280034 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:18.280078 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:36:18.280155 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:36:18.280469 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:36:18.280477 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:36:18.280482 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:36:18.280764 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 12:36:18.281036 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20260619 12:36:18.282337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 10466
I20260619 12:36:18.282451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:36:18.282764 392 external_mini_cluster.cc:1428] Setting key ee45f5ca45cbb15cd84daa9ffe9d5339
I20260619 12:36:18.284430 10594 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33017
I20260619 12:36:18.284708 10594 heartbeater.cc:464] Registering TS with master...
I20260619 12:36:18.285240 10594 heartbeater.cc:511] Master 127.0.98.62:33017 requested a full tablet report, sending...
I20260619 12:36:18.286221 10144 ts_manager.cc:194] Registered new tserver with Master: 29327d85da16443a95c14d751bf63769 (127.0.98.3:39469)
I20260619 12:36:18.297225 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 12:36:18.300371 10183 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:36:18.299337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.0.98.62:33017)
0619 12:36:18.299532 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:18.299536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:18.299680 (+ 144us) server_negotiation.cc:407] Connection header received
0619 12:36:18.299850 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:18.299853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:18.299891 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:18.299986 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:18.300221 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59308: BlockingRecv error: recv got EOF from 127.0.0.1:59308 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:36:18.300491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 10204
I20260619 12:36:18.306145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 10335
I20260619 12:36:18.311576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 10466
I20260619 12:36:18.316850 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 10113
2026-06-19T12:36:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (826 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:36:18 dist-test-slave-kdrb krb5kdc[10604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:36:18 dist-test-slave-kdrb krb5kdc[10604](info): set up 2 sockets
Jun 19 12:36:18 dist-test-slave-kdrb krb5kdc[10604](info): commencing operation
krb5kdc: starting...
W20260619 12:36:19.360498 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:36:19 dist-test-slave-kdrb krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:36:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:36:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:36:19.486266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32949
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:36:19.597273 10617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:19.597550 10617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:19.597591 10617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:19.602174 10617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:36:19.602257 10617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:19.602335 10617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:19.602438 10617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:36:19.602490 10617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:36:19.608564 10617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:19.609834 10617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:19.611045 10617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:19.619717 10623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:19.619709 10622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:19.619709 10625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:19.619836 10617 server_base.cc:1034] running on GCE node
I20260619 12:36:19.620560 10617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:19.621672 10617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:19.622867 10617 hybrid_clock.cc:648] HybridClock initialized: now 1781872579622853 us; error 40 us; skew 500 ppm
Jun 19 12:36:19 dist-test-slave-kdrb krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:36:19.629227 10617 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:36:19.630272 10617 webserver.cc:460] Webserver started at http://127.0.98.62:37617/ using document root <none> and password file <none>
I20260619 12:36:19.630870 10617 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:19.630966 10617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:19.631206 10617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:36:19.633380 10617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "4129c0e7e0cb4c839252429cd7cee2a0"
format_stamp: "Formatted at 2026-06-19 12:36:19 on dist-test-slave-kdrb"
server_key: "fb39f7f30364ef3d2340e1ddd27de3e0"
server_key_iv: "f46509493c5207f424436bc721245b13"
server_key_version: "encryptionkey@0"
I20260619 12:36:19.633941 10617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "4129c0e7e0cb4c839252429cd7cee2a0"
format_stamp: "Formatted at 2026-06-19 12:36:19 on dist-test-slave-kdrb"
server_key: "fb39f7f30364ef3d2340e1ddd27de3e0"
server_key_iv: "f46509493c5207f424436bc721245b13"
server_key_version: "encryptionkey@0"
I20260619 12:36:19.637638 10617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 12:36:19.640092 10632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:19.641346 10617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:36:19.641484 10617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "4129c0e7e0cb4c839252429cd7cee2a0"
format_stamp: "Formatted at 2026-06-19 12:36:19 on dist-test-slave-kdrb"
server_key: "fb39f7f30364ef3d2340e1ddd27de3e0"
server_key_iv: "f46509493c5207f424436bc721245b13"
server_key_version: "encryptionkey@0"
I20260619 12:36:19.641599 10617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:19.655560 10617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:19.658843 10617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:19.659049 10617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:19.666532 10617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32949
I20260619 12:36:19.666821 10684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32949 every 8 connection(s)
I20260619 12:36:19.667635 10617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:36:19.670439 10685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:36:19.672652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 10617
I20260619 12:36:19.672773 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:36:19.673074 392 external_mini_cluster.cc:1428] Setting key d113ddd9294ec517096acbf7f857c9ca
I20260619 12:36:19.677260 10685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0: Bootstrap starting.
I20260619 12:36:19.679627 10685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0: Neither blocks nor log segments found. Creating new log.
I20260619 12:36:19.680380 10685 log.cc:824] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:36:19 dist-test-slave-kdrb krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:36:19.682690 10685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0: No bootstrap required, opened a new log
I20260619 12:36:19.686743 10685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4129c0e7e0cb4c839252429cd7cee2a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } }
I20260619 12:36:19.686991 10685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:36:19.687052 10685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4129c0e7e0cb4c839252429cd7cee2a0, State: Initialized, Role: FOLLOWER
I20260619 12:36:19.687429 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:19.674514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57788 (local address 127.0.98.62:32949)
0619 12:36:19.674941 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:19.674957 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:19.674994 (+ 37us) server_negotiation.cc:407] Connection header received
0619 12:36:19.675730 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:19.675749 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:19.676091 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:19.676436 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:19.677221 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:19.678246 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:19.679128 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:19.679696 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:19.679855 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:19.682582 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:19.682587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:19.682601 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:36:19.682627 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:19.684429 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:19.684938 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:19.684942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:19.684947 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:19.685053 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:19.685356 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:19.685358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:19.685359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:19.685732 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:19.685882 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:19.686190 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 12:36:19.686505 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":123,"threads_started":1}
I20260619 12:36:19.687530 10685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4129c0e7e0cb4c839252429cd7cee2a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } }
I20260619 12:36:19.687721 10685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:36:19.687821 10685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:36:19.687945 10685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:36:19.689083 10685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4129c0e7e0cb4c839252429cd7cee2a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } }
I20260619 12:36:19.689435 10685 leader_election.cc:302] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4129c0e7e0cb4c839252429cd7cee2a0; no voters:
I20260619 12:36:19.689795 10685 leader_election.cc:288] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:36:19.689858 10690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:36:19.690089 10690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [term 1 LEADER]: Becoming Leader. State: Replica: 4129c0e7e0cb4c839252429cd7cee2a0, State: Running, Role: LEADER
I20260619 12:36:19.690455 10690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4129c0e7e0cb4c839252429cd7cee2a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } }
I20260619 12:36:19.691129 10685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:36:19.692255 10691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4129c0e7e0cb4c839252429cd7cee2a0. Latest consensus state: current_term: 1 leader_uuid: "4129c0e7e0cb4c839252429cd7cee2a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4129c0e7e0cb4c839252429cd7cee2a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } } }
I20260619 12:36:19.692278 10692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4129c0e7e0cb4c839252429cd7cee2a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4129c0e7e0cb4c839252429cd7cee2a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32949 } } }
I20260619 12:36:19.692411 10691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [sys.catalog]: This master's current role is: LEADER
I20260619 12:36:19.692420 10692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0 [sys.catalog]: This master's current role is: LEADER
I20260619 12:36:19.692760 10699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:36:19.695858 10699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:36:19.701706 10699 catalog_manager.cc:1347] Generated new cluster ID: fa7383a3ac0e46a1adc10e78e4ec017c
I20260619 12:36:19.701788 10699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:36:19.719511 10699 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:36:19.720503 10699 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:36:19.732033 10699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0: Generated new TSK 0
I20260619 12:36:19.733026 10699 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:36:19.770659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32949
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:36:19.883937 10711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:19.884213 10711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:19.884258 10711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:19.888468 10711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:19.888531 10711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:19.888639 10711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:36:19.894268 10711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:19.895382 10711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:19.896481 10711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:19.906154 10716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:19.906198 10717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:19.906339 10711 server_base.cc:1034] running on GCE node
W20260619 12:36:19.906176 10719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:19.906824 10711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:19.907522 10711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:19.908715 10711 hybrid_clock.cc:648] HybridClock initialized: now 1781872579908700 us; error 33 us; skew 500 ppm
Jun 19 12:36:19 dist-test-slave-kdrb krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:36:19.911566 10711 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:36:19.912413 10711 webserver.cc:460] Webserver started at http://127.0.98.1:34349/ using document root <none> and password file <none>
I20260619 12:36:19.912964 10711 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:19.913051 10711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:19.913275 10711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:36:19.915226 10711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "2746644030da4cdb85a1f2be62b72d97"
format_stamp: "Formatted at 2026-06-19 12:36:19 on dist-test-slave-kdrb"
server_key: "d37fbefc49c446b00996a26f6e94cca6"
server_key_iv: "b5c2f403f2d25d49aa899f6faf6d6fce"
server_key_version: "encryptionkey@0"
I20260619 12:36:19.915833 10711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2746644030da4cdb85a1f2be62b72d97"
format_stamp: "Formatted at 2026-06-19 12:36:19 on dist-test-slave-kdrb"
server_key: "d37fbefc49c446b00996a26f6e94cca6"
server_key_iv: "b5c2f403f2d25d49aa899f6faf6d6fce"
server_key_version: "encryptionkey@0"
I20260619 12:36:19.919463 10711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:36:19.921964 10726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:19.923190 10711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:36:19.923333 10711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "2746644030da4cdb85a1f2be62b72d97"
format_stamp: "Formatted at 2026-06-19 12:36:19 on dist-test-slave-kdrb"
server_key: "d37fbefc49c446b00996a26f6e94cca6"
server_key_iv: "b5c2f403f2d25d49aa899f6faf6d6fce"
server_key_version: "encryptionkey@0"
I20260619 12:36:19.923457 10711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:19.948184 10711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:19.951689 10711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:19.951908 10711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:19.952527 10711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:36:19.953377 10711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:36:19.953447 10711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:19.953539 10711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:36:19.953590 10711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:19.962109 10711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40125
I20260619 12:36:19.962131 10839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40125 every 8 connection(s)
I20260619 12:36:19.963027 10711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:36:19.967000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 10711
I20260619 12:36:19.967134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:36:19.967454 392 external_mini_cluster.cc:1428] Setting key f95594d663ee6c9a23bc884544bee68c
Jun 19 12:36:19 dist-test-slave-kdrb krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:36:19.978449 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:19.964870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51757 (local address 127.0.98.62:32949)
0619 12:36:19.965002 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:19.965006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:19.966087 (+ 1081us) server_negotiation.cc:407] Connection header received
0619 12:36:19.967008 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:19.967011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:19.967069 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:19.967179 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:19.968737 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:19.969312 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:19.971069 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:19.971498 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:19.971674 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:19.974293 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:19.974300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:19.974307 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:36:19.974345 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:19.976214 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:19.976773 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:19.976779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:19.976782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:19.976882 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:19.977229 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:19.977233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:19.977235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:19.977498 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:19.977621 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:19.978127 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 12:36:19.978237 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 12:36:19.979280 10842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:19.965287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.1:51757)
0619 12:36:19.965937 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:19.965979 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:19.966793 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:19.967398 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:19.967409 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:19.967951 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:19.968553 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:19.968568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:19.969450 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:19.969457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:19.970914 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:19.970926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:19.971666 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:19.971669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:19.971842 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:19.972523 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:19.972541 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:19.974092 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:36:19.976361 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:36:19.976367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:36:19.976382 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:36:19.976658 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:36:19.976986 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:36:19.976989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:36:19.976990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:36:19.977133 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:36:19.977605 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:36:19.977610 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:36:19.977874 (+ 264us) client_negotiation.cc:769] Sending connection context
0619 12:36:19.978048 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 12:36:19.978338 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:36:19.980695 10840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32949
I20260619 12:36:19.980940 10840 heartbeater.cc:464] Registering TS with master...
I20260619 12:36:19.981483 10840 heartbeater.cc:511] Master 127.0.98.62:32949 requested a full tablet report, sending...
I20260619 12:36:19.983238 10649 ts_manager.cc:194] Registered new tserver with Master: 2746644030da4cdb85a1f2be62b72d97 (127.0.98.1:40125)
I20260619 12:36:19.984439 10649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:36:19.998291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32949
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:36:20.109965 10845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:20.110224 10845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:20.110298 10845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:20.114491 10845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:20.114560 10845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:20.114660 10845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:36:20.120050 10845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:20.121184 10845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:20.122267 10845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:20.132552 10850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:20.132561 10851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:20.132576 10853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:20.132921 10845 server_base.cc:1034] running on GCE node
I20260619 12:36:20.133354 10845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:20.134042 10845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:20.135239 10845 hybrid_clock.cc:648] HybridClock initialized: now 1781872580135220 us; error 41 us; skew 500 ppm
Jun 19 12:36:20 dist-test-slave-kdrb krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:36:20.138216 10845 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:36:20.139063 10845 webserver.cc:460] Webserver started at http://127.0.98.2:45727/ using document root <none> and password file <none>
I20260619 12:36:20.139675 10845 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:20.139742 10845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:20.139909 10845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:36:20.141903 10845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "4baaeb080d2e441d9f8e0db2eeacfb50"
format_stamp: "Formatted at 2026-06-19 12:36:20 on dist-test-slave-kdrb"
server_key: "14bd5373c297559c9987d19d9f783b6f"
server_key_iv: "9561aaa0576f0c3dbd4d92e3070af57a"
server_key_version: "encryptionkey@0"
I20260619 12:36:20.142392 10845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4baaeb080d2e441d9f8e0db2eeacfb50"
format_stamp: "Formatted at 2026-06-19 12:36:20 on dist-test-slave-kdrb"
server_key: "14bd5373c297559c9987d19d9f783b6f"
server_key_iv: "9561aaa0576f0c3dbd4d92e3070af57a"
server_key_version: "encryptionkey@0"
I20260619 12:36:20.146376 10845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:36:20.148831 10860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:20.150117 10845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:36:20.150225 10845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "4baaeb080d2e441d9f8e0db2eeacfb50"
format_stamp: "Formatted at 2026-06-19 12:36:20 on dist-test-slave-kdrb"
server_key: "14bd5373c297559c9987d19d9f783b6f"
server_key_iv: "9561aaa0576f0c3dbd4d92e3070af57a"
server_key_version: "encryptionkey@0"
I20260619 12:36:20.150362 10845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:20.162184 10845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:20.165557 10845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:20.165809 10845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:20.166411 10845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:36:20.167253 10845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:36:20.167303 10845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:20.167343 10845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:36:20.167358 10845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:20.176154 10845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38787
I20260619 12:36:20.176242 10973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38787 every 8 connection(s)
I20260619 12:36:20.177122 10845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:36:20.184787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 10845
I20260619 12:36:20.184928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:36:20.185268 392 external_mini_cluster.cc:1428] Setting key 3e977959e8bd7fb6b3adfbb7b5521145
Jun 19 12:36:20 dist-test-slave-kdrb krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:36:20.191748 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.178954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52871 (local address 127.0.98.62:32949)
0619 12:36:20.179093 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:20.179097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:20.179963 (+ 866us) server_negotiation.cc:407] Connection header received
0619 12:36:20.181235 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:20.181239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:20.181296 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:20.181414 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:20.182581 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.183112 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.184344 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.184589 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.184700 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:20.187681 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:20.187688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:20.187695 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:36:20.187731 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:20.189365 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:20.190029 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:20.190031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:20.190032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:20.190104 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:20.190497 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:20.190500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:20.190501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:20.190668 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:20.190739 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:20.191452 (+ 713us) server_negotiation.cc:299] Negotiation successful
0619 12:36:20.191595 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:36:20.193181 10976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.179296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.2:52871)
0619 12:36:20.179782 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:20.179839 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:20.181021 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:20.181590 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:20.181597 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:20.181968 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:20.182439 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:20.182453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.183241 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.183250 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:20.184213 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:20.184222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.184742 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.184746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:20.184953 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:20.185614 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:20.185631 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:20.187444 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:36:20.189492 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:36:20.189502 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:36:20.189524 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:36:20.189926 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:36:20.190192 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:36:20.190198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:36:20.190203 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:36:20.190401 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:36:20.190776 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:36:20.190786 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:36:20.191148 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 12:36:20.191379 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 12:36:20.191875 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":136,"threads_started":1}
I20260619 12:36:20.194458 10974 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32949
I20260619 12:36:20.194715 10974 heartbeater.cc:464] Registering TS with master...
I20260619 12:36:20.195205 10974 heartbeater.cc:511] Master 127.0.98.62:32949 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:36:20.196332 10649 ts_manager.cc:194] Registered new tserver with Master: 4baaeb080d2e441d9f8e0db2eeacfb50 (127.0.98.2:38787)
I20260619 12:36:20.196943 10649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:36:20.214257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32949
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:36:20.325362 10979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:36:20.325636 10979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:36:20.325680 10979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:36:20.330240 10979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:36:20.330317 10979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:36:20.330379 10979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:36:20.335810 10979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:36:20.336861 10979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:36:20.337901 10979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:36:20.347777 10984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:20.347810 10987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:36:20.347875 10985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:36:20.347944 10979 server_base.cc:1034] running on GCE node
I20260619 12:36:20.348489 10979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:36:20.349136 10979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:36:20.350297 10979 hybrid_clock.cc:648] HybridClock initialized: now 1781872580350265 us; error 50 us; skew 500 ppm
Jun 19 12:36:20 dist-test-slave-kdrb krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:36:20.353168 10979 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:36:20.354156 10979 webserver.cc:460] Webserver started at http://127.0.98.3:44173/ using document root <none> and password file <none>
I20260619 12:36:20.354753 10979 fs_manager.cc:359] Metadata directory not provided
I20260619 12:36:20.354846 10979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:36:20.355083 10979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:36:20.357179 10979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "d78f46be78ee483881f3056b7f0a22a7"
format_stamp: "Formatted at 2026-06-19 12:36:20 on dist-test-slave-kdrb"
server_key: "19de12486477103d9785b304c97e5e56"
server_key_iv: "58bf3babcf1755a53f9dc9c77cffd8b6"
server_key_version: "encryptionkey@0"
I20260619 12:36:20.357761 10979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d78f46be78ee483881f3056b7f0a22a7"
format_stamp: "Formatted at 2026-06-19 12:36:20 on dist-test-slave-kdrb"
server_key: "19de12486477103d9785b304c97e5e56"
server_key_iv: "58bf3babcf1755a53f9dc9c77cffd8b6"
server_key_version: "encryptionkey@0"
I20260619 12:36:20.361514 10979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 12:36:20.363921 10994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:20.365177 10979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:36:20.365332 10979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "d78f46be78ee483881f3056b7f0a22a7"
format_stamp: "Formatted at 2026-06-19 12:36:20 on dist-test-slave-kdrb"
server_key: "19de12486477103d9785b304c97e5e56"
server_key_iv: "58bf3babcf1755a53f9dc9c77cffd8b6"
server_key_version: "encryptionkey@0"
I20260619 12:36:20.365451 10979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:36:20.381805 10979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:36:20.385210 10979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:36:20.385429 10979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:36:20.386029 10979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:36:20.386849 10979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:36:20.386920 10979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:20.386993 10979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:36:20.387035 10979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:36:20.396310 10979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40421
I20260619 12:36:20.396328 11107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40421 every 8 connection(s)
I20260619 12:36:20.397212 10979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:36:20.400360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 10979
I20260619 12:36:20.400509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:36:20.400786 392 external_mini_cluster.cc:1428] Setting key 33f438624e5d3a17bdaf992ee354747c
Jun 19 12:36:20 dist-test-slave-kdrb krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:36:20.410842 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.399058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53255 (local address 127.0.98.62:32949)
0619 12:36:20.399198 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:20.399202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:20.400142 (+ 940us) server_negotiation.cc:407] Connection header received
0619 12:36:20.401685 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:20.401688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:20.401741 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:20.401814 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:20.403192 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.403797 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.405025 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.405245 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.405361 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:20.407865 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:20.407868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:20.407870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:36:20.407893 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:20.409157 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:20.409602 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:20.409605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:20.409606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:20.409678 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:20.409949 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:20.409951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:20.409952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:20.410110 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:20.410181 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:20.410628 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 12:36:20.410721 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:36:20.411723 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.399455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32949 (local address 127.0.98.3:53255)
0619 12:36:20.399978 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:20.400021 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:20.401454 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:20.401994 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:20.402001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:20.402470 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:36:20.403056 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:20.403069 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.403922 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.403925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:20.404900 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:20.404909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.405399 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.405403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:20.405574 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:20.406164 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:36:20.406180 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:36:20.407707 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:36:20.409260 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:36:20.409265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:36:20.409276 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:36:20.409519 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:36:20.409751 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:36:20.409753 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:36:20.409754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:36:20.409876 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:36:20.410197 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:36:20.410202 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:36:20.410428 (+ 226us) client_negotiation.cc:769] Sending connection context
0619 12:36:20.410585 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 12:36:20.410840 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":116,"threads_started":1}
I20260619 12:36:20.412817 11108 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32949
I20260619 12:36:20.413021 11108 heartbeater.cc:464] Registering TS with master...
I20260619 12:36:20.413550 11108 heartbeater.cc:511] Master 127.0.98.62:32949 requested a full tablet report, sending...
I20260619 12:36:20.414614 10648 ts_manager.cc:194] Registered new tserver with Master: d78f46be78ee483881f3056b7f0a22a7 (127.0.98.3:40421)
I20260619 12:36:20.415205 10648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53255
I20260619 12:36:20.426772 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:36:20.435997 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.428976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.0.98.62:32949)
0619 12:36:20.429104 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:20.429107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:20.429221 (+ 114us) server_negotiation.cc:407] Connection header received
0619 12:36:20.429332 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:20.429334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:20.429376 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:20.429449 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:36:20.430095 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.430603 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.431515 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.431730 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.431811 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:20.433140 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:36:20.433143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:36:20.433145 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:36:20.433169 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:36:20.434456 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:20.434838 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:20.434840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:20.434841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:20.434914 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:36:20.435134 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:36:20.435136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:36:20.435137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:36:20.435356 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:36:20.435456 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:20.435768 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 12:36:20.435875 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 12:36:20.439131 10648 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:36:20.442260 10648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:36:20.457216 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.450739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38787 (local address 127.0.0.1:54696)
0619 12:36:20.451933 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:20.451946 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:20.452029 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:20.452556 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:20.452561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:20.452581 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:36:20.452666 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:20.452674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.454000 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.454003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:20.455638 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:20.455648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.456615 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.456619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:20.456726 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:20.456762 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 12:36:20.456912 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 12:36:20.457033 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"mutex_wait_us":57,"thread_start_us":90,"threads_started":1}
I20260619 12:36:20.457407 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.450702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.0.98.1:40125)
0619 12:36:20.451556 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:20.451571 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:20.451733 (+ 162us) server_negotiation.cc:407] Connection header received
0619 12:36:20.451970 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:20.451976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:20.452195 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:20.452372 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:36:20.452630 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.453855 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.455955 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.456796 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.456907 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:20.457048 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:20.457128 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 12:36:20.457220 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":104,"threads_started":1}
I20260619 12:36:20.457266 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.450554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40125 (local address 127.0.0.1:51882)
0619 12:36:20.451579 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:20.451628 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:20.451785 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:20.452359 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:20.452364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:20.452384 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:36:20.452490 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:20.452495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.454000 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.454003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:20.455838 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:20.455845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.456934 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.456937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:20.457017 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:20.457036 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 12:36:20.457103 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 12:36:20.457158 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":83,"thread_start_us":78,"threads_started":1}
I20260619 12:36:20.457865 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.451369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54696 (local address 127.0.98.2:38787)
0619 12:36:20.452117 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:20.452127 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:20.452147 (+ 20us) server_negotiation.cc:407] Connection header received
0619 12:36:20.452218 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:20.452224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:20.452422 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:20.452590 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:36:20.452751 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.453863 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.455771 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.456487 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.456616 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:20.456775 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:20.457583 (+ 808us) server_negotiation.cc:299] Negotiation successful
0619 12:36:20.457717 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":162,"threads_started":1}
I20260619 12:36:20.459239 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.450864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40421 (local address 127.0.0.1:34998)
0619 12:36:20.454094 (+ 3230us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:20.454129 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:20.454221 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:20.454855 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:20.454858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:20.454875 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:36:20.454979 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:20.454985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.456137 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.456140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:20.457222 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:20.457231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.458852 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.458856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:20.458953 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:20.458976 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:36:20.459052 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 12:36:20.459118 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3180,"thread_start_us":87,"threads_started":1}
I20260619 12:36:20.459328 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.451329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34998 (local address 127.0.98.3:40421)
0619 12:36:20.451802 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:20.451814 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:20.454214 (+ 2400us) server_negotiation.cc:407] Connection header received
0619 12:36:20.454497 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:20.454503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:20.454720 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:20.454880 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:36:20.455072 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.456009 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.457358 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.458545 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.458654 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:20.458780 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:20.459102 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 12:36:20.459196 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":161,"threads_started":1}
I20260619 12:36:20.460696 10908 tablet_service.cc:1467] Processing CreateTablet for tablet e8edf1aab11a409ab2e65fb4626d10f7 (DEFAULT_TABLE table=test-table [id=9e4ce6d028694e90a4543d7706801439]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:36:20.460708 10774 tablet_service.cc:1467] Processing CreateTablet for tablet e8edf1aab11a409ab2e65fb4626d10f7 (DEFAULT_TABLE table=test-table [id=9e4ce6d028694e90a4543d7706801439]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:36:20.461755 10908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8edf1aab11a409ab2e65fb4626d10f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:36:20.461755 10774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8edf1aab11a409ab2e65fb4626d10f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:36:20.461992 11042 tablet_service.cc:1467] Processing CreateTablet for tablet e8edf1aab11a409ab2e65fb4626d10f7 (DEFAULT_TABLE table=test-table [id=9e4ce6d028694e90a4543d7706801439]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:36:20.462883 11042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8edf1aab11a409ab2e65fb4626d10f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:36:20.467801 11125 tablet_bootstrap.cc:492] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50: Bootstrap starting.
I20260619 12:36:20.469372 11126 tablet_bootstrap.cc:492] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97: Bootstrap starting.
I20260619 12:36:20.470691 11125 tablet_bootstrap.cc:654] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50: Neither blocks nor log segments found. Creating new log.
I20260619 12:36:20.471616 11126 tablet_bootstrap.cc:654] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97: Neither blocks nor log segments found. Creating new log.
I20260619 12:36:20.471598 11125 log.cc:824] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50: Log is configured to *not* fsync() on all Append() calls
I20260619 12:36:20.471869 11127 tablet_bootstrap.cc:492] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7: Bootstrap starting.
I20260619 12:36:20.472891 11126 log.cc:824] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97: Log is configured to *not* fsync() on all Append() calls
I20260619 12:36:20.474300 11125 tablet_bootstrap.cc:492] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50: No bootstrap required, opened a new log
I20260619 12:36:20.474484 11127 tablet_bootstrap.cc:654] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7: Neither blocks nor log segments found. Creating new log.
I20260619 12:36:20.474524 11125 ts_tablet_manager.cc:1397] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 12:36:20.474603 11126 tablet_bootstrap.cc:492] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97: No bootstrap required, opened a new log
I20260619 12:36:20.474778 11126 ts_tablet_manager.cc:1397] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 12:36:20.475281 11127 log.cc:824] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7: Log is configured to *not* fsync() on all Append() calls
I20260619 12:36:20.477345 11127 tablet_bootstrap.cc:492] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7: No bootstrap required, opened a new log
I20260619 12:36:20.477536 11127 ts_tablet_manager.cc:1397] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:36:20.477936 11126 raft_consensus.cc:348] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2746644030da4cdb85a1f2be62b72d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "4baaeb080d2e441d9f8e0db2eeacfb50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38787 } } peers { permanent_uuid: "d78f46be78ee483881f3056b7f0a22a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40421 } }
I20260619 12:36:20.478176 11126 raft_consensus.cc:374] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:36:20.478255 11126 raft_consensus.cc:729] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2746644030da4cdb85a1f2be62b72d97, State: Initialized, Role: FOLLOWER
I20260619 12:36:20.478382 11125 raft_consensus.cc:348] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2746644030da4cdb85a1f2be62b72d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "4baaeb080d2e441d9f8e0db2eeacfb50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38787 } } peers { permanent_uuid: "d78f46be78ee483881f3056b7f0a22a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40421 } }
I20260619 12:36:20.478647 11125 raft_consensus.cc:374] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:36:20.478758 11125 raft_consensus.cc:729] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4baaeb080d2e441d9f8e0db2eeacfb50, State: Initialized, Role: FOLLOWER
I20260619 12:36:20.478746 11126 consensus_queue.cc:260] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2746644030da4cdb85a1f2be62b72d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "4baaeb080d2e441d9f8e0db2eeacfb50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38787 } } peers { permanent_uuid: "d78f46be78ee483881f3056b7f0a22a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40421 } }
I20260619 12:36:20.479216 11125 consensus_queue.cc:260] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2746644030da4cdb85a1f2be62b72d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "4baaeb080d2e441d9f8e0db2eeacfb50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38787 } } peers { permanent_uuid: "d78f46be78ee483881f3056b7f0a22a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40421 } }
I20260619 12:36:20.479805 11126 ts_tablet_manager.cc:1428] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260619 12:36:20.480180 10840 heartbeater.cc:503] Master 127.0.98.62:32949 was elected leader, sending a full tablet report...
I20260619 12:36:20.480458 10974 heartbeater.cc:503] Master 127.0.98.62:32949 was elected leader, sending a full tablet report...
I20260619 12:36:20.480499 11125 ts_tablet_manager.cc:1428] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260619 12:36:20.481484 11127 raft_consensus.cc:348] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2746644030da4cdb85a1f2be62b72d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "4baaeb080d2e441d9f8e0db2eeacfb50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38787 } } peers { permanent_uuid: "d78f46be78ee483881f3056b7f0a22a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40421 } }
I20260619 12:36:20.481752 11127 raft_consensus.cc:374] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:36:20.481818 11127 raft_consensus.cc:729] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d78f46be78ee483881f3056b7f0a22a7, State: Initialized, Role: FOLLOWER
I20260619 12:36:20.482249 11127 consensus_queue.cc:260] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2746644030da4cdb85a1f2be62b72d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "4baaeb080d2e441d9f8e0db2eeacfb50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38787 } } peers { permanent_uuid: "d78f46be78ee483881f3056b7f0a22a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40421 } }
I20260619 12:36:20.482893 11108 heartbeater.cc:503] Master 127.0.98.62:32949 was elected leader, sending a full tablet report...
I20260619 12:36:20.483281 11127 ts_tablet_manager.cc:1428] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260619 12:36:20.536516 11131 raft_consensus.cc:482] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:36:20.536715 11131 raft_consensus.cc:504] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2746644030da4cdb85a1f2be62b72d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "4baaeb080d2e441d9f8e0db2eeacfb50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38787 } } peers { permanent_uuid: "d78f46be78ee483881f3056b7f0a22a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40421 } }
I20260619 12:36:20.537884 11131 leader_election.cc:288] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4baaeb080d2e441d9f8e0db2eeacfb50 (127.0.98.2:38787), d78f46be78ee483881f3056b7f0a22a7 (127.0.98.3:40421)
I20260619 12:36:20.542142 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.538234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41871 (local address 127.0.98.2:38787)
0619 12:36:20.538360 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:20.538366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:20.538822 (+ 456us) server_negotiation.cc:407] Connection header received
0619 12:36:20.538905 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:20.538908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:20.538957 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:20.539053 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:36:20.539433 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.540096 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.541113 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.541508 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.541666 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:20.541733 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:20.541921 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 12:36:20.542005 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 12:36:20.542148 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.538296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45413 (local address 127.0.98.3:40421)
0619 12:36:20.538427 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:20.538433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:20.538615 (+ 182us) server_negotiation.cc:407] Connection header received
0619 12:36:20.538722 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:20.538727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:20.538793 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:20.538875 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:36:20.539220 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.539896 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.541058 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.541506 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.541625 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:20.541688 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:20.541974 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 12:36:20.542047 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:36:20.542160 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.538132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38787 (local address 127.0.98.1:41871)
0619 12:36:20.538715 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:20.538729 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:20.538827 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:20.539111 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:20.539113 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:20.539161 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:36:20.539286 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:20.539293 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.540217 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.540220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:20.540962 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:20.540970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.541638 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.541642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:20.541803 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:20.541824 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 12:36:20.541884 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 12:36:20.541982 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":32,"thread_start_us":103,"threads_started":1}
I20260619 12:36:20.542165 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.538124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40421 (local address 127.0.98.1:45413)
0619 12:36:20.538516 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:20.538529 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:36:20.538630 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:36:20.538898 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:36:20.538901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:36:20.538918 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:36:20.539078 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:20.539085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.540010 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.540013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:20.540921 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:36:20.540931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.541641 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.541644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:36:20.541825 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:20.541843 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 12:36:20.541937 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 12:36:20.542011 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":54,"thread_start_us":100,"threads_started":1}
I20260619 12:36:20.542904 11062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8edf1aab11a409ab2e65fb4626d10f7" candidate_uuid: "2746644030da4cdb85a1f2be62b72d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d78f46be78ee483881f3056b7f0a22a7" is_pre_election: true
I20260619 12:36:20.542904 10928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8edf1aab11a409ab2e65fb4626d10f7" candidate_uuid: "2746644030da4cdb85a1f2be62b72d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4baaeb080d2e441d9f8e0db2eeacfb50" is_pre_election: true
I20260619 12:36:20.543241 10928 raft_consensus.cc:2454] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2746644030da4cdb85a1f2be62b72d97 in term 0.
I20260619 12:36:20.543241 11062 raft_consensus.cc:2454] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2746644030da4cdb85a1f2be62b72d97 in term 0.
I20260619 12:36:20.543759 10730 leader_election.cc:302] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2746644030da4cdb85a1f2be62b72d97, 4baaeb080d2e441d9f8e0db2eeacfb50, d78f46be78ee483881f3056b7f0a22a7; no voters:
I20260619 12:36:20.544090 11131 raft_consensus.cc:2781] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:36:20.544283 11131 raft_consensus.cc:482] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:36:20.544399 11131 raft_consensus.cc:3037] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:36:20.546123 11131 raft_consensus.cc:504] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2746644030da4cdb85a1f2be62b72d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "4baaeb080d2e441d9f8e0db2eeacfb50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38787 } } peers { permanent_uuid: "d78f46be78ee483881f3056b7f0a22a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40421 } }
I20260619 12:36:20.546638 11131 leader_election.cc:288] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [CANDIDATE]: Term 1 election: Requested vote from peers 4baaeb080d2e441d9f8e0db2eeacfb50 (127.0.98.2:38787), d78f46be78ee483881f3056b7f0a22a7 (127.0.98.3:40421)
I20260619 12:36:20.547138 11062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8edf1aab11a409ab2e65fb4626d10f7" candidate_uuid: "2746644030da4cdb85a1f2be62b72d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d78f46be78ee483881f3056b7f0a22a7"
I20260619 12:36:20.547286 11062 raft_consensus.cc:3037] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:36:20.547330 10928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8edf1aab11a409ab2e65fb4626d10f7" candidate_uuid: "2746644030da4cdb85a1f2be62b72d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4baaeb080d2e441d9f8e0db2eeacfb50"
I20260619 12:36:20.547459 10928 raft_consensus.cc:3037] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:36:20.548880 11062 raft_consensus.cc:2454] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2746644030da4cdb85a1f2be62b72d97 in term 1.
I20260619 12:36:20.548895 10928 raft_consensus.cc:2454] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2746644030da4cdb85a1f2be62b72d97 in term 1.
I20260619 12:36:20.549230 10729 leader_election.cc:302] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2746644030da4cdb85a1f2be62b72d97, d78f46be78ee483881f3056b7f0a22a7; no voters:
I20260619 12:36:20.549412 11131 raft_consensus.cc:2781] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:36:20.549677 11131 raft_consensus.cc:686] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [term 1 LEADER]: Becoming Leader. State: Replica: 2746644030da4cdb85a1f2be62b72d97, State: Running, Role: LEADER
I20260619 12:36:20.550046 11131 consensus_queue.cc:237] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2746644030da4cdb85a1f2be62b72d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "4baaeb080d2e441d9f8e0db2eeacfb50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38787 } } peers { permanent_uuid: "d78f46be78ee483881f3056b7f0a22a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40421 } }
I20260619 12:36:20.553292 10648 catalog_manager.cc:5495] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2746644030da4cdb85a1f2be62b72d97 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2746644030da4cdb85a1f2be62b72d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2746644030da4cdb85a1f2be62b72d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4baaeb080d2e441d9f8e0db2eeacfb50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d78f46be78ee483881f3056b7f0a22a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40421 } health_report { overall_health: UNKNOWN } } }
I20260619 12:36:20.579450 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.575926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.0.98.1:40125)
0619 12:36:20.576073 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 12:36:20.576077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:36:20.576093 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:36:20.576207 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:36:20.576209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:36:20.576267 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:36:20.576366 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:36:20.576617 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.577122 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.577954 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:36:20.578189 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:36:20.578279 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:36:20.578580 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:36:20.579032 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:36:20.579134 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 12:36:20.579205 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 12:36:20.579259 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 12:36:20.585325 11062 raft_consensus.cc:1261] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7 [term 1 FOLLOWER]: Refusing update from remote peer 2746644030da4cdb85a1f2be62b72d97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:36:20.585325 10928 raft_consensus.cc:1261] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50 [term 1 FOLLOWER]: Refusing update from remote peer 2746644030da4cdb85a1f2be62b72d97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:36:20.586086 11136 consensus_queue.cc:1035] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d78f46be78ee483881f3056b7f0a22a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:36:20.586288 11131 consensus_queue.cc:1035] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4baaeb080d2e441d9f8e0db2eeacfb50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:36:20.603907 10649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:36:20.604064 10649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:36:20.606547 10649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4129c0e7e0cb4c839252429cd7cee2a0: Sending DeleteTablet for 3 replicas of tablet e8edf1aab11a409ab2e65fb4626d10f7
I20260619 12:36:20.607342 10774 tablet_service.cc:1514] Processing DeleteTablet for tablet e8edf1aab11a409ab2e65fb4626d10f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:36:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51882
I20260619 12:36:20.607501 11042 tablet_service.cc:1514] Processing DeleteTablet for tablet e8edf1aab11a409ab2e65fb4626d10f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:36:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34998
I20260619 12:36:20.607527 10908 tablet_service.cc:1514] Processing DeleteTablet for tablet e8edf1aab11a409ab2e65fb4626d10f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:36:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54696
I20260619 12:36:20.608018 11150 tablet_replica.cc:330] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50: stopping tablet replica
I20260619 12:36:20.608104 11149 tablet_replica.cc:330] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97: stopping tablet replica
I20260619 12:36:20.608424 11149 raft_consensus.cc:2229] T e8edf1aab11a409ab2e65fb4626d10f7 P 2746644030da4cdb85a1f2be62b72d97 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:36:20.608771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 10711
I20260619 12:36:20.610556 11150 raft_consensus.cc:2229] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:36:20.610921 11150 raft_consensus.cc:2258] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:36:20.612833 11150 ts_tablet_manager.cc:1905] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:36:20.615577 11150 ts_tablet_manager.cc:1918] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:36:20.615677 11150 log.cc:1196] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/e8edf1aab11a409ab2e65fb4626d10f7
I20260619 12:36:20.615985 11150 ts_tablet_manager.cc:1939] T e8edf1aab11a409ab2e65fb4626d10f7 P 4baaeb080d2e441d9f8e0db2eeacfb50: Deleting consensus metadata
W20260619 12:36:20.616986 10633 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40125 (error 108)
I20260619 12:36:20.617115 11151 tablet_replica.cc:330] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7: stopping tablet replica
I20260619 12:36:20.617445 11151 raft_consensus.cc:2229] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:36:20.617748 11151 raft_consensus.cc:2258] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:36:20.617854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 10845
I20260619 12:36:20.618846 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:36:20.618214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40125 (local address 127.0.0.1:51906)
0619 12:36:20.618686 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 12:36:20.618760 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378}
W20260619 12:36:20.619613 10633 catalog_manager.cc:4584] TS 2746644030da4cdb85a1f2be62b72d97 (127.0.98.1:40125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e8edf1aab11a409ab2e65fb4626d10f7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40125: connect: Connection refused (error 111)
I20260619 12:36:20.621186 11151 ts_tablet_manager.cc:1905] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:36:20.622874 10636 catalog_manager.cc:4841] TS 4baaeb080d2e441d9f8e0db2eeacfb50 (127.0.98.2:38787): tablet e8edf1aab11a409ab2e65fb4626d10f7 (table test-table [id=9e4ce6d028694e90a4543d7706801439]) successfully deleted
I20260619 12:36:20.624343 11151 ts_tablet_manager.cc:1918] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:36:20.624451 11151 log.cc:1196] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/e8edf1aab11a409ab2e65fb4626d10f7
I20260619 12:36:20.624765 11151 ts_tablet_manager.cc:1939] T e8edf1aab11a409ab2e65fb4626d10f7 P d78f46be78ee483881f3056b7f0a22a7: Deleting consensus metadata
I20260619 12:36:20.626400 10635 catalog_manager.cc:4841] TS d78f46be78ee483881f3056b7f0a22a7 (127.0.98.3:40421): tablet e8edf1aab11a409ab2e65fb4626d10f7 (table test-table [id=9e4ce6d028694e90a4543d7706801439]) successfully deleted
I20260619 12:36:20.628758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 10979
I20260619 12:36:20.634748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 10617
2026-06-19T12:36:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2319 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:36:20 dist-test-slave-kdrb krb5kdc[11155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:36:20 dist-test-slave-kdrb krb5kdc[11155](info): set up 2 sockets
Jun 19 12:36:20 dist-test-slave-kdrb krb5kdc[11155](info): commencing operation
krb5kdc: starting...
W20260619 12:36:21.668409 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:36:21 dist-test-slave-kdrb krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:36:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:36:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 12:36:21.839176 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGObR4d/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/postgres -l logfile start
2026-06-19 12:36:24.416 UTC [11185] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 12:36:24.416 UTC [11185] LOG: listening on IPv4 address "127.0.98.20", port 34191
2026-06-19 12:36:24.420 UTC [11185] LOG: listening on Unix socket "/tmp/.s.PGSQL.34191"
2026-06-19 12:36:24.442 UTC [11188] LOG: database system was shut down at 2026-06-19 12:36:22 UTC
2026-06-19 12:36:24.445 UTC [11185] LOG: database system is ready to accept connections
I20260619 12:36:25.400346 392 mini_postgres.cc:96] Postgres bound to 34191
2026-06-19 12:36:25.406 UTC [11196] FATAL: database "slave" does not exist
127.0.98.20:34191 - accepting connections
I20260619 12:36:25.407197 392 mini_ranger.cc:161] Starting Ranger...
I20260619 12:36:25.422189 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 12:36:25.893985 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 12:36:25.894099 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-admin
2026-06-19 12:36:26,155 [I] DB FLAVOR :POSTGRES
2026-06-19 12:36:26,155 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 12:36:26,155 [I] Checking connection..
2026-06-19 12:36:26,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 12:36:26,424 [I] Checking connection passed.
2026-06-19 12:36:26,424 [I] --------- Verifying version history table ---------
2026-06-19 12:36:26,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:36:26,688 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 12:36:26,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:36:26,935 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 12:36:26,936 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 12:36:26,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 12:36:27.184 UTC [11260] WARNING: there is no transaction in progress
2026-06-19 12:36:27,191 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 12:36:27,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:36:27,425 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 12:36:27,425 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 12:36:27,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 12:36:27,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 12:36:27,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kdrb.c.gcp-upstream.internal','N') ;"
2026-06-19 12:36:28,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 12:36:28,406 [I] Table x_portal_user does not exist in database ranger
2026-06-19 12:36:28,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 12:36:28,643 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 12:36:28,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 12:36:28,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 12:36:29,158 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 12:36:29,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 12:36:30.134 UTC [11368] WARNING: there is no transaction in progress
2026-06-19 12:36:30.151 UTC [11368] WARNING: there is no transaction in progress
2026-06-19 12:36:30.168 UTC [11368] WARNING: there is no transaction in progress
2026-06-19 12:36:30.184 UTC [11368] WARNING: there is no transaction in progress
2026-06-19 12:36:30.204 UTC [11368] WARNING: there is no transaction in progress
2026-06-19 12:36:30.222 UTC [11368] WARNING: there is no transaction in progress
2026-06-19 12:36:30.280 UTC [11368] WARNING: there is no transaction in progress
2026-06-19 12:36:30.287 UTC [11368] WARNING: there is no transaction in progress
2026-06-19 12:36:30.303 UTC [11368] WARNING: there is no transaction in progress
2026-06-19 12:36:30.314 UTC [11368] WARNING: there is no transaction in progress
2026-06-19 12:36:30.324 UTC [11368] WARNING: there is no transaction in progress
2026-06-19 12:36:30.338 UTC [11368] WARNING: there is no transaction in progress
2026-06-19 12:36:30.349 UTC [11368] WARNING: there is no transaction in progress
2026-06-19 12:36:30.356 UTC [11368] WARNING: there is no transaction in progress
2026-06-19 12:36:30.363 UTC [11368] WARNING: there is no transaction in progress
2026-06-19 12:36:30.371 UTC [11368] WARNING: there is no transaction in progress
2026-06-19 12:36:31,040 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 12:36:31,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 12:36:31,295 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 12:36:31,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 12:36:31,556 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 12:36:31,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 12:36:31,797 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 12:36:31,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 12:36:32,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 12:36:32,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kdrb.c.gcp-upstream.internal';"
2026-06-19 12:36:32,533 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 12:36:32,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 12:36:32,765 [I] DB_PATCHES have already been applied
I20260619 12:36:32.771616 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGObR4d/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 12:36:32.771684 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 12:36:32.775115 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:58989
Jun 19, 2026 12:36:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 12:36:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 12:36:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 12:36:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 12:36:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 12:36:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 12:36:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 12:36:33.897576 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.490s user 0.000s sys 0.010s
Jun 19, 2026 12:36:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 12:36:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 12:36:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 12:36:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 12:36:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 12:36:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 12:36:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 12:36:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 12:36:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 12:36:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 12:36:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 12:36:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 12:36:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 12:36:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 12:36:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 12:36:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 12:36:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 12:36:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 12:36:56.661046 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 12:36:56 dist-test-slave-kdrb krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872616, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 12:36:56.750845 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 12:36:56.760651 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 12:36:57.677130 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 12:36:57.677235 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-kms
I20260619 12:36:57.677347 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:34191
2026-06-19 12:36:57,867 [I] DB FLAVOR :POSTGRES
2026-06-19 12:36:57,867 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 12:36:57,867 [I] Checking connection
2026-06-19 12:36:57,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 12:36:58,111 [I] connection success
2026-06-19 12:36:58,111 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 12:36:58,111 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 12:36:58,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 12:36:58,347 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 12:36:58,347 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 12:36:58,347 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 12:36:58,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34191/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 12:36:58,627 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 12:36:59.062042 392 mini_ranger_kms.cc:310] Created kms service
I20260619 12:36:59.304600 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 12:36:59.349565 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 12:36:59 dist-test-slave-kdrb krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872616, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:36:59.739950 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 12:36:59.740116 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 12:36:59.740141 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 12:36:59.744911 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11549
I20260619 12:36:59.745090 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:40823
Jun 19, 2026 12:37:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 12:37:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 12:37:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 12:37:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 12:37:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 12:37:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 12:37:02.047704 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.297s user 0.001s sys 0.173s
I20260619 12:37:02.047860 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 12:37:02.047974 392 mini_ranger_kms.cc:401] 127.0.98.20:40823/kms/v1/keys
Jun 19 12:37:02 dist-test-slave-kdrb krb5kdc[11155](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781872622, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 12:37:02 dist-test-slave-kdrb krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872622, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 12:37:02 dist-test-slave-kdrb krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872622, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 12:37:05 dist-test-slave-kdrb krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:37:05.238879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42169
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42169
--ranger_config_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40823/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:37:05.356272 11599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:37:05.356570 11599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:37:05.356611 11599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:37:05.360890 11599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:37:05.360975 11599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:37:05.360996 11599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:37:05.361013 11599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:37:05.361028 11599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:37:05.366963 11599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:40823/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42169
--ranger_config_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:37:05.368187 11599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:37:05.369293 11599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:37:05.378993 11604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:05.379142 11599 server_base.cc:1034] running on GCE node
W20260619 12:37:05.378974 11605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:05.379303 11607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:05.379923 11599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:37:05.381188 11599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:37:05.382382 11599 hybrid_clock.cc:648] HybridClock initialized: now 1781872625382357 us; error 45 us; skew 500 ppm
Jun 19 12:37:05 dist-test-slave-kdrb krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:37:05.394157 11599 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:37:05.395529 11599 webserver.cc:460] Webserver started at http://127.0.98.62:38129/ using document root <none> and password file <none>
I20260619 12:37:05.396157 11599 fs_manager.cc:359] Metadata directory not provided
I20260619 12:37:05.396225 11599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:37:05.396476 11599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:37:05 dist-test-slave-kdrb krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:37:05.452628 11599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "1cea32db73e445de9452d79eb06fe94f"
format_stamp: "Formatted at 2026-06-19 12:37:05 on dist-test-slave-kdrb"
server_key: "b7707cb4e71291a2e6edb5635fbd3424"
server_key_iv: "fe40f2c267bf045903c846fa196ef5a6"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:05.453431 11599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "1cea32db73e445de9452d79eb06fe94f"
format_stamp: "Formatted at 2026-06-19 12:37:05 on dist-test-slave-kdrb"
server_key: "b7707cb4e71291a2e6edb5635fbd3424"
server_key_iv: "fe40f2c267bf045903c846fa196ef5a6"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:05.458603 11599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 12:37:05.476076 11614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:05.477815 11599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:37:05.477981 11599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "1cea32db73e445de9452d79eb06fe94f"
format_stamp: "Formatted at 2026-06-19 12:37:05 on dist-test-slave-kdrb"
server_key: "b7707cb4e71291a2e6edb5635fbd3424"
server_key_iv: "fe40f2c267bf045903c846fa196ef5a6"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:05.478135 11599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:37:05.503501 11599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:37:05.507210 11599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:37:05.507423 11599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:37:05.516386 11599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42169
I20260619 12:37:05.516418 11666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42169 every 8 connection(s)
I20260619 12:37:05.517625 11599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:37:05.520695 11667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:37:05.526419 11667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f: Bootstrap starting.
I20260619 12:37:05.527082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 11599
I20260619 12:37:05.527220 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:37:05.527524 392 external_mini_cluster.cc:1428] Setting key 9d5a569ecd38bb88ccc79f4975971e0e
I20260619 12:37:05.529497 11667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f: Neither blocks nor log segments found. Creating new log.
I20260619 12:37:05.530261 11667 log.cc:824] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f: Log is configured to *not* fsync() on all Append() calls
I20260619 12:37:05.532532 11667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f: No bootstrap required, opened a new log
I20260619 12:37:05.537300 11667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cea32db73e445de9452d79eb06fe94f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42169 } }
I20260619 12:37:05.537638 11667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:37:05.537755 11667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cea32db73e445de9452d79eb06fe94f, State: Initialized, Role: FOLLOWER
I20260619 12:37:05.538326 11667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cea32db73e445de9452d79eb06fe94f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42169 } }
I20260619 12:37:05.538499 11667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:37:05.538583 11667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:37:05.538722 11667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [term 0 FOLLOWER]: Advancing to term 1
Jun 19 12:37:05 dist-test-slave-kdrb krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:37:05.539925 11667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cea32db73e445de9452d79eb06fe94f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42169 } }
I20260619 12:37:05.540356 11667 leader_election.cc:302] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cea32db73e445de9452d79eb06fe94f; no voters:
I20260619 12:37:05.540789 11667 leader_election.cc:288] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:37:05.541854 11672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:37:05.542090 11672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [term 1 LEADER]: Becoming Leader. State: Replica: 1cea32db73e445de9452d79eb06fe94f, State: Running, Role: LEADER
I20260619 12:37:05.542223 11667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:37:05.542451 11672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cea32db73e445de9452d79eb06fe94f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42169 } }
I20260619 12:37:05.544426 11672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cea32db73e445de9452d79eb06fe94f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cea32db73e445de9452d79eb06fe94f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42169 } } }
I20260619 12:37:05.546250 11672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [sys.catalog]: This master's current role is: LEADER
I20260619 12:37:05.546329 11670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:05.528957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.0.98.62:42169)
0619 12:37:05.529433 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:05.529453 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:05.529500 (+ 47us) server_negotiation.cc:407] Connection header received
0619 12:37:05.530373 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:05.530409 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:05.530858 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:05.531263 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:37:05.532151 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:05.533731 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:05.535038 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:05.535798 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:05.535975 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:05.539535 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:37:05.539545 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:37:05.539568 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:37:05.539605 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:37:05.541661 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:05.542184 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:05.542191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:05.542198 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:05.542364 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:05.543391 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:05.543395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:05.543396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:05.543970 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:37:05.544159 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:05.545047 (+ 888us) server_negotiation.cc:299] Negotiation successful
0619 12:37:05.545341 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":149,"threads_started":1}
I20260619 12:37:05.546573 11672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cea32db73e445de9452d79eb06fe94f. Latest consensus state: current_term: 1 leader_uuid: "1cea32db73e445de9452d79eb06fe94f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cea32db73e445de9452d79eb06fe94f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42169 } } }
I20260619 12:37:05.547895 11672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f [sys.catalog]: This master's current role is: LEADER
I20260619 12:37:05.548207 11667 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 12:37:05.549355 11676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:37:05.552637 11676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:37:05.565340 11676 catalog_manager.cc:1347] Generated new cluster ID: 30979f98fa4c4b2993285c8dd4725cd3
I20260619 12:37:05.565431 11676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:37:05.583424 11676 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:37:05.584894 11676 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:37:05.591568 11676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f: Generated new TSK 0
I20260619 12:37:05.592599 11676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:37:06.392189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40823/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42169
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:37:06.549271 11712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:37:06.549794 11712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:37:06.549929 11712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:37:06.556630 11712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:37:06.556743 11712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:37:06.556840 11712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:37:06.565217 11712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:40823/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:37:06.567128 11712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:37:06.568598 11712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:37:06.580040 11720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:06.580052 11722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:06.580040 11719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:06.580976 11712 server_base.cc:1034] running on GCE node
I20260619 12:37:06.581457 11712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:37:06.582170 11712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:37:06.583348 11712 hybrid_clock.cc:648] HybridClock initialized: now 1781872626583322 us; error 55 us; skew 500 ppm
Jun 19 12:37:06 dist-test-slave-kdrb krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:37:06.586400 11712 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:37:06.587386 11712 webserver.cc:460] Webserver started at http://127.0.98.1:46125/ using document root <none> and password file <none>
I20260619 12:37:06.587987 11712 fs_manager.cc:359] Metadata directory not provided
I20260619 12:37:06.588078 11712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:37:06.588297 11712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:37:06 dist-test-slave-kdrb krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:37:06.607683 11712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "a83e05dffb624b8689f2d025f42089e4"
format_stamp: "Formatted at 2026-06-19 12:37:06 on dist-test-slave-kdrb"
server_key: "e73a7853401f7dec7228913e1e202c66"
server_key_iv: "3d120cd221b3b71cfe3ffdfc615d577f"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:06.609903 11712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a83e05dffb624b8689f2d025f42089e4"
format_stamp: "Formatted at 2026-06-19 12:37:06 on dist-test-slave-kdrb"
server_key: "e73a7853401f7dec7228913e1e202c66"
server_key_iv: "3d120cd221b3b71cfe3ffdfc615d577f"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:06.622638 11712 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
I20260619 12:37:06.640972 11731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:06.643177 11712 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:37:06.643342 11712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "a83e05dffb624b8689f2d025f42089e4"
format_stamp: "Formatted at 2026-06-19 12:37:06 on dist-test-slave-kdrb"
server_key: "e73a7853401f7dec7228913e1e202c66"
server_key_iv: "3d120cd221b3b71cfe3ffdfc615d577f"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:06.644105 11712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:37:06.674388 11712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:37:06.678795 11712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:37:06.679085 11712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:37:06.679937 11712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:37:06.681123 11712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:37:06.681236 11712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:06.681372 11712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:37:06.681479 11712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:06.693934 11712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38091
I20260619 12:37:06.695307 11712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:37:06.696138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 11712
I20260619 12:37:06.696264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:37:06.696547 392 external_mini_cluster.cc:1428] Setting key cd1052796a3557c65802bb14340a064c
I20260619 12:37:06.708484 11844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38091 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 19 12:37:06 dist-test-slave-kdrb krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 12:37:06 dist-test-slave-kdrb krb5kdc[11155](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781872626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:37:06.734520 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:06.712511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50299 (local address 127.0.98.62:42169)
0619 12:37:06.717255 (+ 4744us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:06.717262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:06.717281 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:37:06.717327 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:06.717330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:06.717384 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:06.717477 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:37:06.719273 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:06.720170 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:06.722027 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:06.722381 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:06.722485 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:06.729243 (+ 6758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:37:06.729251 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:37:06.729257 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:37:06.729300 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:37:06.731584 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:06.732410 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:06.732415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:06.732419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:06.732526 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:06.733040 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:06.733044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:06.733046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:06.733292 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:37:06.733368 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:06.734178 (+ 810us) server_negotiation.cc:299] Negotiation successful
0619 12:37:06.734332 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4637,"thread_start_us":67,"threads_started":1}
I20260619 12:37:06.736722 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:06.711142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42169 (local address 127.0.98.1:50299)
0619 12:37:06.714923 (+ 3781us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:06.714997 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:06.716289 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:06.717898 (+ 1609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:06.717913 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:06.718296 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:37:06.719068 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:06.719090 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:06.720354 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:06.720357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:06.721871 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:06.721884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:06.722749 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:06.722756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:06.722991 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:06.726855 (+ 3864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:37:06.726887 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:37:06.728928 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:37:06.731791 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:06.731801 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:06.731828 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:06.732229 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:06.732664 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:06.732670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:06.732674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:06.732910 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:06.733453 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:37:06.733462 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:37:06.733875 (+ 413us) client_negotiation.cc:769] Sending connection context
0619 12:37:06.735062 (+ 1187us) client_negotiation.cc:240] Negotiation successful
0619 12:37:06.735523 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3490,"thread_start_us":166,"threads_started":1}
I20260619 12:37:06.738857 11845 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42169
I20260619 12:37:06.739132 11845 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:06.739905 11845 heartbeater.cc:511] Master 127.0.98.62:42169 requested a full tablet report, sending...
I20260619 12:37:06.741940 11631 ts_manager.cc:194] Registered new tserver with Master: a83e05dffb624b8689f2d025f42089e4 (127.0.98.1:38091)
I20260619 12:37:06.743673 11631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:37:06.759603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40823/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42169
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:37:06.941926 11851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:37:06.942404 11851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:37:06.942525 11851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:37:06.948774 11851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:37:06.948930 11851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:37:06.949066 11851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:37:06.957504 11851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:40823/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:37:06.958806 11851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:37:06.959998 11851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:37:06.973641 11858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:06.973977 11860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:06.974319 11851 server_base.cc:1034] running on GCE node
W20260619 12:37:06.974512 11857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:06.975088 11851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:37:06.976063 11851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:37:06.983526 11851 hybrid_clock.cc:648] HybridClock initialized: now 1781872626983485 us; error 73 us; skew 500 ppm
Jun 19 12:37:06 dist-test-slave-kdrb krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:37:06.987069 11851 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:37:06.988340 11851 webserver.cc:460] Webserver started at http://127.0.98.2:46165/ using document root <none> and password file <none>
I20260619 12:37:06.989089 11851 fs_manager.cc:359] Metadata directory not provided
I20260619 12:37:06.989174 11851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:37:06.989473 11851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:37:06 dist-test-slave-kdrb krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:37:07.006635 11851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "fb7f2c798cb24f5199a6f540d7c28d6e"
format_stamp: "Formatted at 2026-06-19 12:37:07 on dist-test-slave-kdrb"
server_key: "7380df39d350320296f9eec81919eda9"
server_key_iv: "db2bc0b743d7a308f81b8687873702af"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:07.007385 11851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fb7f2c798cb24f5199a6f540d7c28d6e"
format_stamp: "Formatted at 2026-06-19 12:37:07 on dist-test-slave-kdrb"
server_key: "7380df39d350320296f9eec81919eda9"
server_key_iv: "db2bc0b743d7a308f81b8687873702af"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:07.012879 11851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 12:37:07.038738 11867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:07.040498 11851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:37:07.040647 11851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "fb7f2c798cb24f5199a6f540d7c28d6e"
format_stamp: "Formatted at 2026-06-19 12:37:07 on dist-test-slave-kdrb"
server_key: "7380df39d350320296f9eec81919eda9"
server_key_iv: "db2bc0b743d7a308f81b8687873702af"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:07.040792 11851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:37:07.068073 11851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:37:07.071907 11851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:37:07.072156 11851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:37:07.072911 11851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:37:07.073994 11851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:37:07.074100 11851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:07.074200 11851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:37:07.074261 11851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:07.088538 11851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38749
I20260619 12:37:07.089828 11851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:37:07.099035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 11851
I20260619 12:37:07.099169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:37:07.099455 392 external_mini_cluster.cc:1428] Setting key 59aaf513f97a1828bcd3c4e23333c783
I20260619 12:37:07.091347 11980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38749 every 8 connection(s)
Jun 19 12:37:07 dist-test-slave-kdrb krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:37:07.110158 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.095166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51093 (local address 127.0.98.62:42169)
0619 12:37:07.095291 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:07.095296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:07.095834 (+ 538us) server_negotiation.cc:407] Connection header received
0619 12:37:07.097186 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:07.097190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:07.097245 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:07.097298 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:37:07.098919 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.099784 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.102284 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.102643 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.102718 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:07.105842 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:37:07.105847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:37:07.105850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:37:07.105881 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:37:07.107679 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:07.108301 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:07.108305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:07.108307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:07.108394 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:07.108796 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:07.108799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:07.108801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:07.109019 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:37:07.109089 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:07.109827 (+ 738us) server_negotiation.cc:299] Negotiation successful
0619 12:37:07.109964 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:37:07.112090 11983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.094012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42169 (local address 127.0.98.2:51093)
0619 12:37:07.095666 (+ 1654us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:07.095727 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:07.096857 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:07.097720 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:07.097729 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:07.098093 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:37:07.098753 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:07.098771 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.100027 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.100030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:07.102144 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:07.102155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.102918 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.102921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:07.103119 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:07.103811 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:37:07.103834 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:37:07.105621 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:37:07.107816 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:07.107822 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:07.107841 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:07.108186 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:07.108499 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:07.108502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:07.108504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:07.108689 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:07.109173 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:37:07.109180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:37:07.109530 (+ 350us) client_negotiation.cc:769] Sending connection context
0619 12:37:07.110618 (+ 1088us) client_negotiation.cc:240] Negotiation successful
0619 12:37:07.110984 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1396,"thread_start_us":145,"threads_started":1}
I20260619 12:37:07.113750 11981 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42169
I20260619 12:37:07.114027 11981 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:07.114622 11981 heartbeater.cc:511] Master 127.0.98.62:42169 requested a full tablet report, sending...
I20260619 12:37:07.115979 11631 ts_manager.cc:194] Registered new tserver with Master: fb7f2c798cb24f5199a6f540d7c28d6e (127.0.98.2:38749)
I20260619 12:37:07.116751 11631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51093
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:37:07.149147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40823/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42169
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:37:07 dist-test-slave-kdrb krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 12:37:07.312801 11986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:37:07.313182 11986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:37:07.313244 11986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:37:07.319114 11986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:37:07.319195 11986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:37:07.319278 11986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:37:07.326994 11986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:40823/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:37:07.328433 11986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:37:07.329773 11986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:37:07.342406 11994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:07.342341 11992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:07.342834 11986 server_base.cc:1034] running on GCE node
W20260619 12:37:07.342605 11991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:07.344077 11986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:37:07.344893 11986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:37:07.346084 11986 hybrid_clock.cc:648] HybridClock initialized: now 1781872627346054 us; error 50 us; skew 500 ppm
Jun 19 12:37:07 dist-test-slave-kdrb krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:37:07.350291 11986 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:37:07.351308 11986 webserver.cc:460] Webserver started at http://127.0.98.3:34401/ using document root <none> and password file <none>
I20260619 12:37:07.351953 11986 fs_manager.cc:359] Metadata directory not provided
I20260619 12:37:07.352043 11986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:37:07.352274 11986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:37:07 dist-test-slave-kdrb krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:37:07.365232 11986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "1f72f380958f4f97815556f14af825f9"
format_stamp: "Formatted at 2026-06-19 12:37:07 on dist-test-slave-kdrb"
server_key: "a571959b5b6efafabd24e00d17bcbdc8"
server_key_iv: "a03c7f4adafed3dd46ad090aa38502f7"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:07.366176 11986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1f72f380958f4f97815556f14af825f9"
format_stamp: "Formatted at 2026-06-19 12:37:07 on dist-test-slave-kdrb"
server_key: "a571959b5b6efafabd24e00d17bcbdc8"
server_key_iv: "a03c7f4adafed3dd46ad090aa38502f7"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:07.373185 11986 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.004s
I20260619 12:37:07.393054 12001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:07.394846 11986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 12:37:07.394981 11986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "1f72f380958f4f97815556f14af825f9"
format_stamp: "Formatted at 2026-06-19 12:37:07 on dist-test-slave-kdrb"
server_key: "a571959b5b6efafabd24e00d17bcbdc8"
server_key_iv: "a03c7f4adafed3dd46ad090aa38502f7"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:07.395118 11986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:37:07.423362 11986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:37:07.426642 11986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:37:07.426867 11986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:37:07.427567 11986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:37:07.428512 11986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:37:07.428582 11986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:07.428687 11986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:37:07.428743 11986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:07.440824 11986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45493
I20260619 12:37:07.440840 12114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45493 every 8 connection(s)
I20260619 12:37:07.441911 11986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:37:07.451763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 11986
I20260619 12:37:07.451906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:37:07.452221 392 external_mini_cluster.cc:1428] Setting key 8f5bbfb17144d0d0970eca273d9697e2
Jun 19 12:37:07 dist-test-slave-kdrb krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:37:07.461926 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.445589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51211 (local address 127.0.98.62:42169)
0619 12:37:07.445745 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:07.445749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:07.448550 (+ 2801us) server_negotiation.cc:407] Connection header received
0619 12:37:07.449739 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:07.449752 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:07.449811 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:07.449864 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:37:07.451535 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.452340 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.453980 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.454337 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.454427 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:07.457543 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:37:07.457547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:37:07.457550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:37:07.457579 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:37:07.459360 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:07.460082 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:07.460086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:07.460088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:07.460174 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:07.460615 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:07.460618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:07.460620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:07.460844 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:37:07.460932 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:07.461640 (+ 708us) server_negotiation.cc:299] Negotiation successful
0619 12:37:07.461767 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 12:37:07.463063 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.444469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42169 (local address 127.0.98.3:51211)
0619 12:37:07.448341 (+ 3872us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:07.448404 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:07.449463 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:07.450301 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:07.450310 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:07.450663 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:37:07.451350 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:07.451368 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.452496 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.452499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:07.453840 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:07.453851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.454627 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.454630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:07.454835 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:07.455584 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:37:07.455603 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:37:07.457334 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:37:07.459526 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:07.459533 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:07.459552 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:07.459955 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:07.460279 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:07.460285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:07.460289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:07.460501 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:07.460965 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:37:07.460973 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:37:07.461347 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 12:37:07.461570 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 12:37:07.461938 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3636,"thread_start_us":151,"threads_started":1}
I20260619 12:37:07.464457 12115 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42169
I20260619 12:37:07.464723 12115 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:07.465299 12115 heartbeater.cc:511] Master 127.0.98.62:42169 requested a full tablet report, sending...
I20260619 12:37:07.466477 11630 ts_manager.cc:194] Registered new tserver with Master: 1f72f380958f4f97815556f14af825f9 (127.0.98.3:45493)
I20260619 12:37:07.467120 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:37:07.467252 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51211
I20260619 12:37:07.481508 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.469406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.0.98.62:42169)
0619 12:37:07.469519 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:07.469523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:07.471795 (+ 2272us) server_negotiation.cc:407] Connection header received
0619 12:37:07.471915 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:07.471918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:07.471972 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:07.472072 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:37:07.472901 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.473691 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.474907 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.475229 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.475346 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:07.477626 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:37:07.477633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:37:07.477640 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:37:07.477679 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:37:07.479536 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:07.480101 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:07.480105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:07.480107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:07.480197 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:07.480493 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:07.480496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:07.480498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:07.480706 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:37:07.480841 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:07.481201 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 12:37:07.481334 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:37:07.486007 11630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:37:07.490872 11630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:37:07.520048 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.508382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38749 (local address 127.0.0.1:54420)
0619 12:37:07.510521 (+ 2139us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:07.510531 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:07.510590 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:07.515615 (+ 5025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:07.515621 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:07.515647 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:37:07.515797 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:07.515807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.517219 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.517222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:07.518369 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:07.518380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.519504 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.519508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:07.519628 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:07.519670 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 12:37:07.519772 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 12:37:07.519900 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2088,"thread_start_us":81,"threads_started":1}
I20260619 12:37:07.520713 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.508983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38091 (local address 127.0.0.1:36412)
0619 12:37:07.510723 (+ 1740us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:07.510734 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:07.510808 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:07.511526 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:07.511533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:07.511555 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:37:07.511678 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:07.511684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.513058 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.513061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:07.514466 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:07.514477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.520324 (+ 5847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.520329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:07.520429 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:07.520454 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 12:37:07.520517 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 12:37:07.520584 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1698,"thread_start_us":80,"threads_started":1}
I20260619 12:37:07.521111 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.509124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.0.98.2:38749)
0619 12:37:07.514848 (+ 5724us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:07.514861 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:07.514886 (+ 25us) server_negotiation.cc:407] Connection header received
0619 12:37:07.514965 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:07.514971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:07.515205 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:07.515401 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:37:07.515942 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.517085 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.518541 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.519349 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.520658 (+ 1309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:07.520828 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:07.520882 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 12:37:07.520977 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5559,"thread_start_us":112,"threads_started":1}
I20260619 12:37:07.521508 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.509337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.0.98.1:38091)
0619 12:37:07.509901 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:07.509908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:07.510891 (+ 983us) server_negotiation.cc:407] Connection header received
0619 12:37:07.510972 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:07.510977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:07.511201 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:07.511342 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:37:07.511815 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.512926 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.514613 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.519310 (+ 4697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.519421 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:07.521202 (+ 1781us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:07.521270 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 12:37:07.521372 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":95,"threads_started":1}
I20260619 12:37:07.524437 11779 tablet_service.cc:1467] Processing CreateTablet for tablet c2f69313495a46ca8987ad95ea6cfc1f (DEFAULT_TABLE table=test-table [id=cca87d95b03644b1a1a5593fa0292c61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:37:07.524441 11915 tablet_service.cc:1467] Processing CreateTablet for tablet c2f69313495a46ca8987ad95ea6cfc1f (DEFAULT_TABLE table=test-table [id=cca87d95b03644b1a1a5593fa0292c61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:37:07.525561 11915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f69313495a46ca8987ad95ea6cfc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:37:07.525561 11779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f69313495a46ca8987ad95ea6cfc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:37:07.532789 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.507269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45493 (local address 127.0.0.1:53004)
0619 12:37:07.510150 (+ 2881us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:07.510205 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:07.510326 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:07.528275 (+ 17949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:07.528280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:07.528304 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:37:07.528421 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:07.528431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.530025 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.530029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:07.531174 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:07.531185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.532385 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.532389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:07.532480 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:07.532509 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 12:37:07.532563 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 12:37:07.532641 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2706,"thread_start_us":101,"threads_started":1}
I20260619 12:37:07.533669 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.507419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53004 (local address 127.0.98.3:45493)
0619 12:37:07.527726 (+ 20307us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:07.527738 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:07.527770 (+ 32us) server_negotiation.cc:407] Connection header received
0619 12:37:07.527856 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:07.527862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:07.528074 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:07.528598 (+ 524us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:37:07.528745 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.529877 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.531366 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.532204 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.533207 (+ 1003us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:07.533387 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:07.533438 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 12:37:07.533528 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20174,"thread_start_us":179,"threads_started":1}
I20260619 12:37:07.534473 12134 tablet_bootstrap.cc:492] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e: Bootstrap starting.
I20260619 12:37:07.537439 12134 tablet_bootstrap.cc:654] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e: Neither blocks nor log segments found. Creating new log.
I20260619 12:37:07.538434 12134 log.cc:824] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:37:07.539357 12135 tablet_bootstrap.cc:492] T c2f69313495a46ca8987ad95ea6cfc1f P a83e05dffb624b8689f2d025f42089e4: Bootstrap starting.
Jun 19 12:37:07 dist-test-slave-kdrb krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:37:07.542667 12135 tablet_bootstrap.cc:654] T c2f69313495a46ca8987ad95ea6cfc1f P a83e05dffb624b8689f2d025f42089e4: Neither blocks nor log segments found. Creating new log.
I20260619 12:37:07.543757 12135 log.cc:824] T c2f69313495a46ca8987ad95ea6cfc1f P a83e05dffb624b8689f2d025f42089e4: Log is configured to *not* fsync() on all Append() calls
I20260619 12:37:07.544389 12049 tablet_service.cc:1467] Processing CreateTablet for tablet c2f69313495a46ca8987ad95ea6cfc1f (DEFAULT_TABLE table=test-table [id=cca87d95b03644b1a1a5593fa0292c61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:37:07.546109 12135 tablet_bootstrap.cc:492] T c2f69313495a46ca8987ad95ea6cfc1f P a83e05dffb624b8689f2d025f42089e4: No bootstrap required, opened a new log
I20260619 12:37:07.546353 12135 ts_tablet_manager.cc:1397] T c2f69313495a46ca8987ad95ea6cfc1f P a83e05dffb624b8689f2d025f42089e4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 12:37:07.547151 12049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f69313495a46ca8987ad95ea6cfc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:37:07.550901 12135 raft_consensus.cc:348] T c2f69313495a46ca8987ad95ea6cfc1f P a83e05dffb624b8689f2d025f42089e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f72f380958f4f97815556f14af825f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45493 } } peers { permanent_uuid: "fb7f2c798cb24f5199a6f540d7c28d6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38749 } } peers { permanent_uuid: "a83e05dffb624b8689f2d025f42089e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38091 } }
I20260619 12:37:07.551194 12135 raft_consensus.cc:374] T c2f69313495a46ca8987ad95ea6cfc1f P a83e05dffb624b8689f2d025f42089e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:37:07.551244 12135 raft_consensus.cc:729] T c2f69313495a46ca8987ad95ea6cfc1f P a83e05dffb624b8689f2d025f42089e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a83e05dffb624b8689f2d025f42089e4, State: Initialized, Role: FOLLOWER
I20260619 12:37:07.553156 12134 tablet_bootstrap.cc:492] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e: No bootstrap required, opened a new log
I20260619 12:37:07.553376 12134 ts_tablet_manager.cc:1397] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20260619 12:37:07.557891 12134 raft_consensus.cc:348] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f72f380958f4f97815556f14af825f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45493 } } peers { permanent_uuid: "fb7f2c798cb24f5199a6f540d7c28d6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38749 } } peers { permanent_uuid: "a83e05dffb624b8689f2d025f42089e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38091 } }
I20260619 12:37:07.558276 12134 raft_consensus.cc:374] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:37:07.558378 12134 raft_consensus.cc:729] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb7f2c798cb24f5199a6f540d7c28d6e, State: Initialized, Role: FOLLOWER
I20260619 12:37:07.558938 12134 consensus_queue.cc:260] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f72f380958f4f97815556f14af825f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45493 } } peers { permanent_uuid: "fb7f2c798cb24f5199a6f540d7c28d6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38749 } } peers { permanent_uuid: "a83e05dffb624b8689f2d025f42089e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38091 } }
I20260619 12:37:07.559406 12135 consensus_queue.cc:260] T c2f69313495a46ca8987ad95ea6cfc1f P a83e05dffb624b8689f2d025f42089e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f72f380958f4f97815556f14af825f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45493 } } peers { permanent_uuid: "fb7f2c798cb24f5199a6f540d7c28d6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38749 } } peers { permanent_uuid: "a83e05dffb624b8689f2d025f42089e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38091 } }
I20260619 12:37:07.560585 12134 ts_tablet_manager.cc:1428] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 12:37:07.561556 12135 ts_tablet_manager.cc:1428] T c2f69313495a46ca8987ad95ea6cfc1f P a83e05dffb624b8689f2d025f42089e4: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260619 12:37:07.562024 11981 heartbeater.cc:503] Master 127.0.98.62:42169 was elected leader, sending a full tablet report...
I20260619 12:37:07.563133 11845 heartbeater.cc:503] Master 127.0.98.62:42169 was elected leader, sending a full tablet report...
I20260619 12:37:07.566470 12140 tablet_bootstrap.cc:492] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9: Bootstrap starting.
I20260619 12:37:07.570386 12140 tablet_bootstrap.cc:654] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9: Neither blocks nor log segments found. Creating new log.
I20260619 12:37:07.571365 12140 log.cc:824] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9: Log is configured to *not* fsync() on all Append() calls
I20260619 12:37:07.573894 12140 tablet_bootstrap.cc:492] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9: No bootstrap required, opened a new log
I20260619 12:37:07.574106 12140 ts_tablet_manager.cc:1397] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260619 12:37:07.578684 12140 raft_consensus.cc:348] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f72f380958f4f97815556f14af825f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45493 } } peers { permanent_uuid: "fb7f2c798cb24f5199a6f540d7c28d6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38749 } } peers { permanent_uuid: "a83e05dffb624b8689f2d025f42089e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38091 } }
I20260619 12:37:07.579058 12140 raft_consensus.cc:374] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:37:07.579171 12140 raft_consensus.cc:729] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f72f380958f4f97815556f14af825f9, State: Initialized, Role: FOLLOWER
I20260619 12:37:07.579720 12140 consensus_queue.cc:260] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f72f380958f4f97815556f14af825f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45493 } } peers { permanent_uuid: "fb7f2c798cb24f5199a6f540d7c28d6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38749 } } peers { permanent_uuid: "a83e05dffb624b8689f2d025f42089e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38091 } }
I20260619 12:37:07.580575 12115 heartbeater.cc:503] Master 127.0.98.62:42169 was elected leader, sending a full tablet report...
I20260619 12:37:07.582338 12140 ts_tablet_manager.cc:1428] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260619 12:37:07.592344 11982 tablet.cc:2369] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:37:07.671422 12138 raft_consensus.cc:482] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:37:07.671700 12138 raft_consensus.cc:504] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f72f380958f4f97815556f14af825f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45493 } } peers { permanent_uuid: "fb7f2c798cb24f5199a6f540d7c28d6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38749 } } peers { permanent_uuid: "a83e05dffb624b8689f2d025f42089e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38091 } }
I20260619 12:37:07.673163 12138 leader_election.cc:288] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f72f380958f4f97815556f14af825f9 (127.0.98.3:45493), a83e05dffb624b8689f2d025f42089e4 (127.0.98.1:38091)
I20260619 12:37:07.686012 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.674313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60823 (local address 127.0.98.3:45493)
0619 12:37:07.674459 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:07.674464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:07.680066 (+ 5602us) server_negotiation.cc:407] Connection header received
0619 12:37:07.680364 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:07.680368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:07.680421 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:07.680468 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:37:07.681191 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.681918 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.684716 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.685298 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.685380 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:07.685449 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:07.685791 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 12:37:07.685867 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 12:37:07.686798 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.673415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45493 (local address 127.0.98.2:60823)
0619 12:37:07.679944 (+ 6529us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:07.679965 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:07.680219 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:07.680609 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:07.680612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:07.680631 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:37:07.680744 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:07.680750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.682095 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.682099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:07.683424 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:07.683434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.685545 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.685549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:07.685656 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:07.685679 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:37:07.686546 (+ 867us) client_negotiation.cc:240] Negotiation successful
0619 12:37:07.686628 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6420,"thread_start_us":93,"threads_started":1}
I20260619 12:37:07.687493 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.674455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44647 (local address 127.0.98.1:38091)
0619 12:37:07.674568 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:07.674573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:07.681029 (+ 6456us) server_negotiation.cc:407] Connection header received
0619 12:37:07.681145 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:07.681151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:07.681206 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:07.681278 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:37:07.682655 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.683409 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.686102 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.686739 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.686828 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:07.686906 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:07.687271 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 12:37:07.687356 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 12:37:07.687942 12069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f69313495a46ca8987ad95ea6cfc1f" candidate_uuid: "fb7f2c798cb24f5199a6f540d7c28d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f72f380958f4f97815556f14af825f9" is_pre_election: true
I20260619 12:37:07.688295 12069 raft_consensus.cc:2454] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb7f2c798cb24f5199a6f540d7c28d6e in term 0.
I20260619 12:37:07.689117 11869 leader_election.cc:302] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f72f380958f4f97815556f14af825f9, fb7f2c798cb24f5199a6f540d7c28d6e; no voters:
I20260619 12:37:07.689467 12138 raft_consensus.cc:2781] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:37:07.689543 12138 raft_consensus.cc:482] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:37:07.689587 12138 raft_consensus.cc:3037] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:37:07.690989 12138 raft_consensus.cc:504] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f72f380958f4f97815556f14af825f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45493 } } peers { permanent_uuid: "fb7f2c798cb24f5199a6f540d7c28d6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38749 } } peers { permanent_uuid: "a83e05dffb624b8689f2d025f42089e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38091 } }
I20260619 12:37:07.691398 12138 leader_election.cc:288] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [CANDIDATE]: Term 1 election: Requested vote from peers 1f72f380958f4f97815556f14af825f9 (127.0.98.3:45493), a83e05dffb624b8689f2d025f42089e4 (127.0.98.1:38091)
I20260619 12:37:07.692050 12069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f69313495a46ca8987ad95ea6cfc1f" candidate_uuid: "fb7f2c798cb24f5199a6f540d7c28d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f72f380958f4f97815556f14af825f9"
I20260619 12:37:07.692188 12069 raft_consensus.cc:3037] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:37:07.693567 12069 raft_consensus.cc:2454] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb7f2c798cb24f5199a6f540d7c28d6e in term 1.
W20260619 12:37:07.694115 12116 tablet.cc:2369] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:37:07.694808 11869 leader_election.cc:302] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f72f380958f4f97815556f14af825f9, fb7f2c798cb24f5199a6f540d7c28d6e; no voters:
I20260619 12:37:07.695053 12138 raft_consensus.cc:2781] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:37:07.695281 12138 raft_consensus.cc:686] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [term 1 LEADER]: Becoming Leader. State: Replica: fb7f2c798cb24f5199a6f540d7c28d6e, State: Running, Role: LEADER
I20260619 12:37:07.695698 12138 consensus_queue.cc:237] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f72f380958f4f97815556f14af825f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45493 } } peers { permanent_uuid: "fb7f2c798cb24f5199a6f540d7c28d6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38749 } } peers { permanent_uuid: "a83e05dffb624b8689f2d025f42089e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38091 } }
I20260619 12:37:07.699517 11631 catalog_manager.cc:5495] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e reported cstate change: term changed from 0 to 1, leader changed from <none> to fb7f2c798cb24f5199a6f540d7c28d6e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fb7f2c798cb24f5199a6f540d7c28d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f72f380958f4f97815556f14af825f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb7f2c798cb24f5199a6f540d7c28d6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a83e05dffb624b8689f2d025f42089e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38091 } health_report { overall_health: UNKNOWN } } }
I20260619 12:37:07.700073 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.673846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38091 (local address 127.0.98.2:44647)
0619 12:37:07.680948 (+ 7102us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:07.680962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:07.681058 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:07.682390 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:07.682394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:07.682412 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:37:07.682510 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:07.682516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.683640 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.683643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:07.685962 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:07.685971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.686994 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.686997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:07.687125 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:07.687155 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 12:37:07.699747 (+ 12592us) client_negotiation.cc:240] Negotiation successful
0619 12:37:07.699865 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7037,"thread_start_us":96,"threads_started":1}
I20260619 12:37:07.701395 11799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f69313495a46ca8987ad95ea6cfc1f" candidate_uuid: "fb7f2c798cb24f5199a6f540d7c28d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a83e05dffb624b8689f2d025f42089e4" is_pre_election: true
I20260619 12:37:07.701752 11799 raft_consensus.cc:2454] T c2f69313495a46ca8987ad95ea6cfc1f P a83e05dffb624b8689f2d025f42089e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb7f2c798cb24f5199a6f540d7c28d6e in term 0.
I20260619 12:37:07.702150 11798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f69313495a46ca8987ad95ea6cfc1f" candidate_uuid: "fb7f2c798cb24f5199a6f540d7c28d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a83e05dffb624b8689f2d025f42089e4"
I20260619 12:37:07.702268 11798 raft_consensus.cc:3037] T c2f69313495a46ca8987ad95ea6cfc1f P a83e05dffb624b8689f2d025f42089e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:37:07.703637 11798 raft_consensus.cc:2454] T c2f69313495a46ca8987ad95ea6cfc1f P a83e05dffb624b8689f2d025f42089e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb7f2c798cb24f5199a6f540d7c28d6e in term 1.
W20260619 12:37:07.713030 11846 tablet.cc:2369] T c2f69313495a46ca8987ad95ea6cfc1f P a83e05dffb624b8689f2d025f42089e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:37:07.732857 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.727692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.0.98.2:38749)
0619 12:37:07.727895 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:07.727900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:07.727914 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:37:07.728003 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:07.728005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:07.728072 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:07.728191 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:37:07.728540 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.729329 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.730559 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:07.730906 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:07.731000 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:07.731421 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:37:07.732050 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:37:07.732174 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:07.732630 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 12:37:07.732707 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260619 12:37:07.740178 12069 raft_consensus.cc:1261] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9 [term 1 FOLLOWER]: Refusing update from remote peer fb7f2c798cb24f5199a6f540d7c28d6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:37:07.741160 11798 raft_consensus.cc:1261] T c2f69313495a46ca8987ad95ea6cfc1f P a83e05dffb624b8689f2d025f42089e4 [term 1 FOLLOWER]: Refusing update from remote peer fb7f2c798cb24f5199a6f540d7c28d6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:37:07.742193 12146 consensus_queue.cc:1035] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a83e05dffb624b8689f2d025f42089e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:37:07.742367 12138 consensus_queue.cc:1035] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f72f380958f4f97815556f14af825f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:37:07.758888 12149 mvcc.cc:205] Tried to move back new op lower bound from 7298550283212787712 to 7298550283045744640. Current Snapshot: MvccSnapshot[applied={T|T < 7298550283212787712}]
I20260619 12:37:07.764526 12150 mvcc.cc:205] Tried to move back new op lower bound from 7298550283212787712 to 7298550283045744640. Current Snapshot: MvccSnapshot[applied={T|T < 7298550283212787712}]
I20260619 12:37:07.767936 12152 mvcc.cc:205] Tried to move back new op lower bound from 7298550283212787712 to 7298550283045744640. Current Snapshot: MvccSnapshot[applied={T|T < 7298550283212787712}]
I20260619 12:37:07.775194 11631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:37:07.775394 11631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:37:07.778690 11631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cea32db73e445de9452d79eb06fe94f: Sending DeleteTablet for 3 replicas of tablet c2f69313495a46ca8987ad95ea6cfc1f
I20260619 12:37:07.779790 12049 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f69313495a46ca8987ad95ea6cfc1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:37:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53004
I20260619 12:37:07.780807 11779 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f69313495a46ca8987ad95ea6cfc1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:37:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36412
I20260619 12:37:07.781798 11915 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f69313495a46ca8987ad95ea6cfc1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:37:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54420
I20260619 12:37:07.782980 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 11712
I20260619 12:37:07.783550 12158 tablet_replica.cc:330] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9: stopping tablet replica
I20260619 12:37:07.783905 12158 raft_consensus.cc:2229] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:37:07.791850 12158 raft_consensus.cc:2258] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:37:07.799625 12158 ts_tablet_manager.cc:1905] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:37:07.800038 12160 tablet_replica.cc:330] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e: stopping tablet replica
I20260619 12:37:07.800853 12160 raft_consensus.cc:2229] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:37:07.801983 12160 raft_consensus.cc:2258] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:37:07.804373 12158 ts_tablet_manager.cc:1918] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:37:07.804803 12158 log.cc:1196] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/c2f69313495a46ca8987ad95ea6cfc1f
I20260619 12:37:07.806038 12160 ts_tablet_manager.cc:1905] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:37:07.807896 12158 ts_tablet_manager.cc:1939] T c2f69313495a46ca8987ad95ea6cfc1f P 1f72f380958f4f97815556f14af825f9: Deleting consensus metadata
I20260619 12:37:07.808651 12160 ts_tablet_manager.cc:1918] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:37:07.808754 12160 log.cc:1196] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/c2f69313495a46ca8987ad95ea6cfc1f
I20260619 12:37:07.809105 12160 ts_tablet_manager.cc:1939] T c2f69313495a46ca8987ad95ea6cfc1f P fb7f2c798cb24f5199a6f540d7c28d6e: Deleting consensus metadata
I20260619 12:37:07.809513 11616 catalog_manager.cc:4841] TS 1f72f380958f4f97815556f14af825f9 (127.0.98.3:45493): tablet c2f69313495a46ca8987ad95ea6cfc1f (table test-table [id=cca87d95b03644b1a1a5593fa0292c61]) successfully deleted
W20260619 12:37:07.809914 11615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38091 (error 108)
I20260619 12:37:07.810382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 11851
I20260619 12:37:07.810600 11615 catalog_manager.cc:4841] TS fb7f2c798cb24f5199a6f540d7c28d6e (127.0.98.2:38749): tablet c2f69313495a46ca8987ad95ea6cfc1f (table test-table [id=cca87d95b03644b1a1a5593fa0292c61]) successfully deleted
I20260619 12:37:07.812772 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:07.812450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38091 (local address 127.0.0.1:36420)
0619 12:37:07.812604 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:07.812676 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 12:37:07.813051 11615 catalog_manager.cc:4584] TS a83e05dffb624b8689f2d025f42089e4 (127.0.98.1:38091): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c2f69313495a46ca8987ad95ea6cfc1f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38091: connect: Connection refused (error 111)
I20260619 12:37:07.817292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 11986
I20260619 12:37:07.824370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 11599
I20260619 12:37:07.830996 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 12:37:08.176296 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 12:37:08.176425 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 12:37:08.521667 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 12:37:08.521 UTC [11185] LOG: received smart shutdown request
2026-06-19 12:37:08.525 UTC [11185] LOG: background worker "logical replication launcher" (PID 11194) exited with exit code 1
2026-06-19 12:37:08.526 UTC [11189] LOG: shutting down
2026-06-19 12:37:08.701 UTC [11185] LOG: database system is shut down
2026-06-19T12:37:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48183 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:37:08 dist-test-slave-kdrb krb5kdc[12170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:37:08 dist-test-slave-kdrb krb5kdc[12170](info): set up 2 sockets
Jun 19 12:37:08 dist-test-slave-kdrb krb5kdc[12170](info): commencing operation
krb5kdc: starting...
W20260619 12:37:09.867585 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:37:09 dist-test-slave-kdrb krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:37:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:37:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 12:37:10.044332 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGObR4d/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/postgres -l logfile start
2026-06-19 12:37:12.676 UTC [12200] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 12:37:12.676 UTC [12200] LOG: listening on IPv4 address "127.0.98.20", port 45845
2026-06-19 12:37:12.681 UTC [12200] LOG: listening on Unix socket "/tmp/.s.PGSQL.45845"
2026-06-19 12:37:12.716 UTC [12203] LOG: database system was shut down at 2026-06-19 12:37:11 UTC
2026-06-19 12:37:12.721 UTC [12200] LOG: database system is ready to accept connections
I20260619 12:37:13.659425 392 mini_postgres.cc:96] Postgres bound to 45845
2026-06-19 12:37:13.665 UTC [12211] FATAL: database "slave" does not exist
127.0.98.20:45845 - accepting connections
I20260619 12:37:13.666554 392 mini_ranger.cc:161] Starting Ranger...
I20260619 12:37:13.682226 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 12:37:14.175602 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 12:37:14.175676 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-admin
2026-06-19 12:37:14,471 [I] DB FLAVOR :POSTGRES
2026-06-19 12:37:14,471 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 12:37:14,472 [I] Checking connection..
2026-06-19 12:37:14,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 12:37:14,745 [I] Checking connection passed.
2026-06-19 12:37:14,745 [I] --------- Verifying version history table ---------
2026-06-19 12:37:14,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:37:15,001 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 12:37:15,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:37:15,261 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 12:37:15,261 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 12:37:15,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 12:37:15.507 UTC [12275] WARNING: there is no transaction in progress
2026-06-19 12:37:15,514 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 12:37:15,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:37:15,782 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 12:37:15,782 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 12:37:15,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 12:37:16,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 12:37:16,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kdrb.c.gcp-upstream.internal','N') ;"
2026-06-19 12:37:16,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 12:37:16,793 [I] Table x_portal_user does not exist in database ranger
2026-06-19 12:37:16,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 12:37:17,061 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 12:37:17,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 12:37:17,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 12:37:17,560 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 12:37:17,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 12:37:18.521 UTC [12383] WARNING: there is no transaction in progress
2026-06-19 12:37:18.538 UTC [12383] WARNING: there is no transaction in progress
2026-06-19 12:37:18.556 UTC [12383] WARNING: there is no transaction in progress
2026-06-19 12:37:18.571 UTC [12383] WARNING: there is no transaction in progress
2026-06-19 12:37:18.586 UTC [12383] WARNING: there is no transaction in progress
2026-06-19 12:37:18.604 UTC [12383] WARNING: there is no transaction in progress
2026-06-19 12:37:18.671 UTC [12383] WARNING: there is no transaction in progress
2026-06-19 12:37:18.678 UTC [12383] WARNING: there is no transaction in progress
2026-06-19 12:37:18.692 UTC [12383] WARNING: there is no transaction in progress
2026-06-19 12:37:18.702 UTC [12383] WARNING: there is no transaction in progress
2026-06-19 12:37:18.713 UTC [12383] WARNING: there is no transaction in progress
2026-06-19 12:37:18.727 UTC [12383] WARNING: there is no transaction in progress
2026-06-19 12:37:18.738 UTC [12383] WARNING: there is no transaction in progress
2026-06-19 12:37:18.746 UTC [12383] WARNING: there is no transaction in progress
2026-06-19 12:37:18.754 UTC [12383] WARNING: there is no transaction in progress
2026-06-19 12:37:18.761 UTC [12383] WARNING: there is no transaction in progress
2026-06-19 12:37:19,431 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 12:37:19,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 12:37:19,675 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 12:37:19,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 12:37:19,913 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 12:37:19,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 12:37:20,159 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 12:37:20,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 12:37:20,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 12:37:20,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kdrb.c.gcp-upstream.internal';"
2026-06-19 12:37:20,914 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 12:37:20,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 12:37:21,143 [I] DB_PATCHES have already been applied
I20260619 12:37:21.149292 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGObR4d/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 12:37:21.149369 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 12:37:21.152927 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:46307
Jun 19, 2026 12:37:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 12:37:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 12:37:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 12:37:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 12:37:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 12:37:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 12:37:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 12:37:22.266554 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.600s user 0.002s sys 0.008s
Jun 19, 2026 12:37:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 12:37:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 12:37:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 12:37:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 12:37:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 12:37:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 12:37:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 12:37:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 12:37:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 12:37:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 12:37:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 12:37:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 12:37:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 12:37:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 12:37:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 12:37:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 12:37:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 12:37:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 12:37:44.691412 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 12:37:44 dist-test-slave-kdrb krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872664, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 12:37:44.777776 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 12:37:44.787587 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 12:37:45.646401 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 12:37:45.646500 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-kms
I20260619 12:37:45.646522 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:45845
2026-06-19 12:37:45,823 [I] DB FLAVOR :POSTGRES
2026-06-19 12:37:45,823 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 12:37:45,823 [I] Checking connection
2026-06-19 12:37:45,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 12:37:46,080 [I] connection success
2026-06-19 12:37:46,080 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 12:37:46,080 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 12:37:46,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 12:37:46,335 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 12:37:46,336 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 12:37:46,336 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 12:37:46,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45845/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 12:37:46,597 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 12:37:47.049031 392 mini_ranger_kms.cc:310] Created kms service
I20260619 12:37:47.277702 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 12:37:47.322341 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 12:37:47 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872664, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:37:47.754781 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 12:37:47.754954 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGObR4d/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGObR4d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 12:37:47.754976 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 12:37:47.759359 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12563
I20260619 12:37:47.759464 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:56413
Jun 19, 2026 12:37:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 12:37:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 12:37:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 12:37:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 12:37:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 12:37:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 12:37:50.077574 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.300s user 0.008s sys 0.152s
I20260619 12:37:50.077726 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 12:37:50.077821 392 mini_ranger_kms.cc:401] 127.0.98.20:56413/kms/v1/keys
Jun 19 12:37:50 dist-test-slave-kdrb krb5kdc[12170](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781872670, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 12:37:50 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872670, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 12:37:51 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872670, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 12:37:53 dist-test-slave-kdrb krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:37:53.550987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40057
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40057
--ranger_config_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56413/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:37:53.686306 12614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:37:53.686607 12614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:37:53.686646 12614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:37:53.690761 12614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:37:53.690832 12614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:37:53.690851 12614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:37:53.690867 12614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:37:53.690883 12614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:37:53.696609 12614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:56413/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40057
--ranger_config_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:37:53.697770 12614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:37:53.698870 12614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:37:53.708539 12622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:53.708489 12620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:53.708508 12614 server_base.cc:1034] running on GCE node
W20260619 12:37:53.708549 12619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:53.709462 12614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:37:53.710588 12614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:37:53.711782 12614 hybrid_clock.cc:648] HybridClock initialized: now 1781872673711748 us; error 51 us; skew 500 ppm
Jun 19 12:37:53 dist-test-slave-kdrb krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:37:53.719426 12614 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:37:53.720487 12614 webserver.cc:460] Webserver started at http://127.0.98.62:41845/ using document root <none> and password file <none>
I20260619 12:37:53.721073 12614 fs_manager.cc:359] Metadata directory not provided
I20260619 12:37:53.721132 12614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:37:53.721370 12614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:37:53 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:37:53.772027 12614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "45d023ad17dc4167969f3ba6eb3368a4"
format_stamp: "Formatted at 2026-06-19 12:37:53 on dist-test-slave-kdrb"
server_key: "a09d4c23a3b170538ad953648fb323ce"
server_key_iv: "e9bb6131db1cc00e052a8d9b22de0c64"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:53.772825 12614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "45d023ad17dc4167969f3ba6eb3368a4"
format_stamp: "Formatted at 2026-06-19 12:37:53 on dist-test-slave-kdrb"
server_key: "a09d4c23a3b170538ad953648fb323ce"
server_key_iv: "e9bb6131db1cc00e052a8d9b22de0c64"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:53.777755 12614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 12:37:53.797406 12629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:53.799351 12614 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 12:37:53.799561 12614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "45d023ad17dc4167969f3ba6eb3368a4"
format_stamp: "Formatted at 2026-06-19 12:37:53 on dist-test-slave-kdrb"
server_key: "a09d4c23a3b170538ad953648fb323ce"
server_key_iv: "e9bb6131db1cc00e052a8d9b22de0c64"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:53.799732 12614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:37:53.824301 12614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:37:53.827035 12614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:37:53.827193 12614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:37:53.834685 12614 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40057
I20260619 12:37:53.834692 12681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40057 every 8 connection(s)
I20260619 12:37:53.835722 12614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:37:53.838552 12682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:37:53.840268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 12614
I20260619 12:37:53.840413 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:37:53.840708 392 external_mini_cluster.cc:1428] Setting key 8ab76609899b5a79a0f3794ea59909e4
I20260619 12:37:53.845099 12682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4: Bootstrap starting.
I20260619 12:37:53.847723 12682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4: Neither blocks nor log segments found. Creating new log.
I20260619 12:37:53.848659 12682 log.cc:824] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:37:53 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:37:53.850970 12682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4: No bootstrap required, opened a new log
I20260619 12:37:53.854496 12682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d023ad17dc4167969f3ba6eb3368a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } }
I20260619 12:37:53.854815 12682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:37:53.854910 12682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d023ad17dc4167969f3ba6eb3368a4, State: Initialized, Role: FOLLOWER
I20260619 12:37:53.855448 12682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d023ad17dc4167969f3ba6eb3368a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } }
I20260619 12:37:53.855638 12682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:37:53.855703 12682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:37:53.855808 12682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:37:53.856462 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:53.842097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40594 (local address 127.0.98.62:40057)
0619 12:37:53.842510 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:53.842525 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:53.842560 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:37:53.843223 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:53.843240 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:53.843626 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:53.843909 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:37:53.844814 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:53.845834 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:53.847028 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:53.847539 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:53.847653 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:53.850623 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:37:53.850628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:37:53.850641 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:37:53.850669 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:37:53.852950 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:53.853487 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:53.853492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:53.853497 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:53.853674 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:53.854015 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:53.854021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:53.854026 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:53.854639 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:37:53.854858 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:53.855313 (+ 455us) server_negotiation.cc:299] Negotiation successful
0619 12:37:53.855653 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":137,"threads_started":1}
I20260619 12:37:53.856784 12682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d023ad17dc4167969f3ba6eb3368a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } }
I20260619 12:37:53.857127 12682 leader_election.cc:302] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45d023ad17dc4167969f3ba6eb3368a4; no voters:
I20260619 12:37:53.857460 12682 leader_election.cc:288] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:37:53.857550 12687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:37:53.857756 12687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 1 LEADER]: Becoming Leader. State: Replica: 45d023ad17dc4167969f3ba6eb3368a4, State: Running, Role: LEADER
I20260619 12:37:53.858108 12687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d023ad17dc4167969f3ba6eb3368a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } }
I20260619 12:37:53.858804 12682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:37:53.859921 12689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45d023ad17dc4167969f3ba6eb3368a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d023ad17dc4167969f3ba6eb3368a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } } }
I20260619 12:37:53.860056 12689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [sys.catalog]: This master's current role is: LEADER
I20260619 12:37:53.860191 12688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45d023ad17dc4167969f3ba6eb3368a4. Latest consensus state: current_term: 1 leader_uuid: "45d023ad17dc4167969f3ba6eb3368a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d023ad17dc4167969f3ba6eb3368a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } } }
I20260619 12:37:53.860364 12688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [sys.catalog]: This master's current role is: LEADER
I20260619 12:37:53.860605 12696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:37:53.863941 12696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:37:53.864369 12682 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 12:37:53.873284 12696 catalog_manager.cc:1347] Generated new cluster ID: c18ecf2673de43cebaa6d7ec5c5fc50e
I20260619 12:37:53.873368 12696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:37:53.902258 12696 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:37:53.903862 12696 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:37:53.909586 12696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4: Generated new TSK 0
I20260619 12:37:53.910552 12696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:37:54.595856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40057
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:37:54.720099 12727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:37:54.720430 12727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:37:54.720547 12727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:37:54.725113 12727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:37:54.725206 12727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:37:54.725307 12727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:37:54.733847 12727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:56413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:37:54.735622 12727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:37:54.737224 12727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:37:54.748567 12737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:54.748694 12735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:54.748559 12734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:54.749717 12727 server_base.cc:1034] running on GCE node
I20260619 12:37:54.750219 12727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:37:54.751026 12727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:37:54.752200 12727 hybrid_clock.cc:648] HybridClock initialized: now 1781872674752169 us; error 57 us; skew 500 ppm
Jun 19 12:37:54 dist-test-slave-kdrb krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:37:54.755184 12727 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:37:54.756296 12727 webserver.cc:460] Webserver started at http://127.0.98.1:42047/ using document root <none> and password file <none>
I20260619 12:37:54.756901 12727 fs_manager.cc:359] Metadata directory not provided
I20260619 12:37:54.756995 12727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:37:54.757234 12727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:37:54 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:37:54.773742 12727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "c0280ae0fd564958ab77e7fc26b9e972"
format_stamp: "Formatted at 2026-06-19 12:37:54 on dist-test-slave-kdrb"
server_key: "4c79e727e9a0d4d4d749e53c0480530f"
server_key_iv: "102874b611ab5f6311f5c8f867947cfe"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:54.776101 12727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c0280ae0fd564958ab77e7fc26b9e972"
format_stamp: "Formatted at 2026-06-19 12:37:54 on dist-test-slave-kdrb"
server_key: "4c79e727e9a0d4d4d749e53c0480530f"
server_key_iv: "102874b611ab5f6311f5c8f867947cfe"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:54.785097 12727 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.004s
I20260619 12:37:54.803951 12744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:54.805738 12727 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 12:37:54.805876 12727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "c0280ae0fd564958ab77e7fc26b9e972"
format_stamp: "Formatted at 2026-06-19 12:37:54 on dist-test-slave-kdrb"
server_key: "4c79e727e9a0d4d4d749e53c0480530f"
server_key_iv: "102874b611ab5f6311f5c8f867947cfe"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:54.806007 12727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:37:54.837713 12727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:37:54.842001 12727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:37:54.842299 12727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:37:54.843129 12727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:37:54.844338 12727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:37:54.844448 12727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:54.844554 12727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:37:54.844614 12727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:54.857336 12727 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43095
I20260619 12:37:54.857510 12857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43095 every 8 connection(s)
I20260619 12:37:54.858778 12727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:37:54.860059 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 12727
I20260619 12:37:54.860176 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:37:54.860455 392 external_mini_cluster.cc:1428] Setting key 6653cd0dc38afefefd63cf162eaa7925
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 19 12:37:54 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:37:54.887737 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:54.861991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50759 (local address 127.0.98.62:40057)
0619 12:37:54.864230 (+ 2239us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:54.864240 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:54.864257 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:37:54.865353 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:54.865357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:54.865422 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:54.865489 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:37:54.867293 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:54.868265 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:54.870194 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:54.870572 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:54.870711 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:54.882852 (+ 12141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:37:54.882859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:37:54.882862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:37:54.882901 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:37:54.885029 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:54.885796 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:54.885799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:54.885801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:54.885905 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:54.886348 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:54.886353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:54.886355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:54.886562 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:37:54.886673 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:54.887357 (+ 684us) server_negotiation.cc:299] Negotiation successful
0619 12:37:54.887527 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":116,"threads_started":1}
I20260619 12:37:54.888630 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:54.862574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40057 (local address 127.0.98.1:50759)
0619 12:37:54.863797 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:54.863858 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:54.865077 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:54.865952 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:54.865964 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:54.866342 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:37:54.867087 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:54.867113 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:54.868449 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:54.868454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:54.870025 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:54.870040 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:54.870775 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:54.870780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:54.871015 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:54.876369 (+ 5354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:37:54.876398 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:37:54.882473 (+ 6075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:37:54.885180 (+ 2707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:54.885195 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:54.885211 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:54.885659 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:54.886031 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:54.886034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:54.886036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:54.886242 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:54.886674 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:37:54.886682 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:37:54.887057 (+ 375us) client_negotiation.cc:769] Sending connection context
0619 12:37:54.887267 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 12:37:54.887654 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"thread_start_us":129,"threads_started":1}
I20260619 12:37:54.890417 12858 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40057
I20260619 12:37:54.890709 12858 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:54.891350 12858 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
I20260619 12:37:54.893291 12646 ts_manager.cc:194] Registered new tserver with Master: c0280ae0fd564958ab77e7fc26b9e972 (127.0.98.1:43095)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:37:54.894851 12646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50759
I20260619 12:37:54.895509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40057
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:37:54 dist-test-slave-kdrb krb5kdc[12170](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781872674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 12:37:55.028107 12866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:37:55.028432 12866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:37:55.028515 12866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:37:55.033179 12866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:37:55.033272 12866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:37:55.033375 12866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:37:55.039955 12866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:56413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:37:55.041412 12866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:37:55.042542 12866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:37:55.053289 12872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:55.053609 12873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:55.053428 12875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:55.053890 12866 server_base.cc:1034] running on GCE node
I20260619 12:37:55.054391 12866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:37:55.055145 12866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:37:55.056351 12866 hybrid_clock.cc:648] HybridClock initialized: now 1781872675056293 us; error 73 us; skew 500 ppm
Jun 19 12:37:55 dist-test-slave-kdrb krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:37:55.060374 12866 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:37:55.061604 12866 webserver.cc:460] Webserver started at http://127.0.98.2:35023/ using document root <none> and password file <none>
I20260619 12:37:55.062354 12866 fs_manager.cc:359] Metadata directory not provided
I20260619 12:37:55.062448 12866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:37:55.062685 12866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:37:55 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:37:55.078297 12866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "2a16a85af5504fe5ab553f2c5ef74e0d"
format_stamp: "Formatted at 2026-06-19 12:37:55 on dist-test-slave-kdrb"
server_key: "9c55eec14a4a64255a7d81e0e762f57d"
server_key_iv: "139201c54a0c1f34d1b5fb434028e6cc"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:55.079434 12866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2a16a85af5504fe5ab553f2c5ef74e0d"
format_stamp: "Formatted at 2026-06-19 12:37:55 on dist-test-slave-kdrb"
server_key: "9c55eec14a4a64255a7d81e0e762f57d"
server_key_iv: "139201c54a0c1f34d1b5fb434028e6cc"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:55.093474 12866 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.000s
I20260619 12:37:55.112067 12882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:55.113922 12866 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:37:55.114069 12866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "2a16a85af5504fe5ab553f2c5ef74e0d"
format_stamp: "Formatted at 2026-06-19 12:37:55 on dist-test-slave-kdrb"
server_key: "9c55eec14a4a64255a7d81e0e762f57d"
server_key_iv: "139201c54a0c1f34d1b5fb434028e6cc"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:55.114207 12866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:37:55.138510 12866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:37:55.142472 12866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:37:55.142696 12866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:37:55.143498 12866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:37:55.144704 12866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:37:55.144804 12866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:55.144884 12866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:37:55.144929 12866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:55.156658 12866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34795
I20260619 12:37:55.158025 12866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:37:55.160959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 12866
I20260619 12:37:55.161104 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:37:55.161464 392 external_mini_cluster.cc:1428] Setting key b67fc4eb60604e0f7057abcacd48df57
I20260619 12:37:55.168818 12995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34795 every 8 connection(s)
Jun 19 12:37:55 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:37:55.181391 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:55.164128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39963 (local address 127.0.98.62:40057)
0619 12:37:55.164288 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:55.164295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:55.164847 (+ 552us) server_negotiation.cc:407] Connection header received
0619 12:37:55.166006 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:55.166010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:55.166064 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:55.166147 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:37:55.167931 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:55.168852 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:55.170954 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:55.171458 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:55.171609 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:55.175190 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:37:55.175197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:37:55.175200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:37:55.175236 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:37:55.177150 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:55.177804 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:55.177808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:55.177810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:55.177903 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:55.178398 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:55.178404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:55.178408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:55.178633 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:37:55.178702 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:55.181020 (+ 2318us) server_negotiation.cc:299] Negotiation successful
0619 12:37:55.181195 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:37:55.182994 12998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:55.161838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40057 (local address 127.0.98.2:39963)
0619 12:37:55.164657 (+ 2819us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:55.164740 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:55.165775 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:55.166471 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:55.166484 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:55.166894 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:37:55.167730 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:55.167756 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:55.169038 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:55.169043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:55.170739 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:55.170762 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:55.171881 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:55.171886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:55.172130 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:55.172989 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:37:55.173023 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:37:55.174917 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:37:55.177288 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:55.177296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:55.177316 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:55.177688 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:55.178040 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:55.178043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:55.178045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:55.178273 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:55.180171 (+ 1898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:37:55.180180 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:37:55.180662 (+ 482us) client_negotiation.cc:769] Sending connection context
0619 12:37:55.181671 (+ 1009us) client_negotiation.cc:240] Negotiation successful
0619 12:37:55.182009 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2557,"thread_start_us":181,"threads_started":1}
I20260619 12:37:55.184811 12996 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40057
I20260619 12:37:55.185092 12996 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:55.185665 12996 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
I20260619 12:37:55.187003 12646 ts_manager.cc:194] Registered new tserver with Master: 2a16a85af5504fe5ab553f2c5ef74e0d (127.0.98.2:34795)
I20260619 12:37:55.187853 12646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:37:55.199119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40057
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:37:55.342008 13001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:37:55.342336 13001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:37:55.342463 13001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:37:55.347244 13001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:37:55.347343 13001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:37:55.347452 13001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:37:55.354393 13001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:56413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:37:55.355815 13001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:37:55.357033 13001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:37:55.367885 13007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:55.368238 13006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:55.368489 13009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:55.369462 13001 server_base.cc:1034] running on GCE node
I20260619 12:37:55.369999 13001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:37:55.370884 13001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:37:55.372097 13001 hybrid_clock.cc:648] HybridClock initialized: now 1781872675372071 us; error 43 us; skew 500 ppm
Jun 19 12:37:55 dist-test-slave-kdrb krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:37:55.376078 13001 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:37:55.377262 13001 webserver.cc:460] Webserver started at http://127.0.98.3:45229/ using document root <none> and password file <none>
I20260619 12:37:55.378007 13001 fs_manager.cc:359] Metadata directory not provided
I20260619 12:37:55.378103 13001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:37:55.378350 13001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:37:55 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:37:55.400956 13001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "5b4e25a32ce745d894d14876c0478f16"
format_stamp: "Formatted at 2026-06-19 12:37:55 on dist-test-slave-kdrb"
server_key: "4deda2bec5d1d94e93d481c2b468680e"
server_key_iv: "89007633a847185b7f3619ee0a09b113"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:55.409499 13001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5b4e25a32ce745d894d14876c0478f16"
format_stamp: "Formatted at 2026-06-19 12:37:55 on dist-test-slave-kdrb"
server_key: "4deda2bec5d1d94e93d481c2b468680e"
server_key_iv: "89007633a847185b7f3619ee0a09b113"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:55.415831 13001 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260619 12:37:55.436334 13016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:55.438203 13001 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260619 12:37:55.438349 13001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "5b4e25a32ce745d894d14876c0478f16"
format_stamp: "Formatted at 2026-06-19 12:37:55 on dist-test-slave-kdrb"
server_key: "4deda2bec5d1d94e93d481c2b468680e"
server_key_iv: "89007633a847185b7f3619ee0a09b113"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:55.438478 13001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:37:55.481087 13001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Jun 19 12:37:55 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:37:55.485481 13001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:37:55.485751 13001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:37:55.486521 13001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:37:55.487783 13001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:37:55.487885 13001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:55.487982 13001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:37:55.488047 13001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:55.499783 13001 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42051
I20260619 12:37:55.501116 13001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:37:55.504047 13129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42051 every 8 connection(s)
I20260619 12:37:55.508178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 13001
I20260619 12:37:55.508320 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:37:55.508627 392 external_mini_cluster.cc:1428] Setting key 67c78894effbf364b9feabe89e424224
Jun 19 12:37:55 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:37:55.527786 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:55.509234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58607 (local address 127.0.98.62:40057)
0619 12:37:55.509679 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:55.509685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:55.513299 (+ 3614us) server_negotiation.cc:407] Connection header received
0619 12:37:55.514496 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:55.514499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:55.514553 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:55.514626 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:37:55.516107 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:55.516782 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:55.518500 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:55.518831 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:55.518909 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:55.521885 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:37:55.521889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:37:55.521891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:37:55.521921 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:37:55.523754 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:55.524361 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:55.524365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:55.524367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:55.524452 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:55.525395 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:55.525399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:55.525401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:55.525636 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:37:55.525719 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:55.527449 (+ 1730us) server_negotiation.cc:299] Negotiation successful
0619 12:37:55.527626 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350}
I20260619 12:37:55.529656 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:55.509597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40057 (local address 127.0.98.3:58607)
0619 12:37:55.513133 (+ 3536us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:55.513184 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:55.514272 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:55.514816 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:55.514826 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:55.515217 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:37:55.515909 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:55.515931 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:55.516909 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:55.516913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:55.518307 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:55.518318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:55.519115 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:55.519119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:55.519333 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:55.520020 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:37:55.520037 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:37:55.521696 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:37:55.523882 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:55.523889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:55.523906 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:55.524238 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:55.524685 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:55.524688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:55.524690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:55.525281 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:55.525786 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:37:55.525792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:37:55.527168 (+ 1376us) client_negotiation.cc:769] Sending connection context
0619 12:37:55.528232 (+ 1064us) client_negotiation.cc:240] Negotiation successful
0619 12:37:55.528594 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2002,"thread_start_us":194,"threads_started":1}
I20260619 12:37:55.531116 13130 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40057
I20260619 12:37:55.531399 13130 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:55.532042 13130 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
I20260619 12:37:55.533221 12646 ts_manager.cc:194] Registered new tserver with Master: 5b4e25a32ce745d894d14876c0478f16 (127.0.98.3:42051)
I20260619 12:37:55.533967 12646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58607
I20260619 12:37:55.535065 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:37:55.535939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 12727
I20260619 12:37:55.545490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 12866
I20260619 12:37:55.554522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 13001
I20260619 12:37:55.562316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 12614
I20260619 12:37:55.570909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40057
--webserver_interface=127.0.98.62
--webserver_port=41845
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40057
--ranger_config_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56413/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56413/kms with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:37:55.700543 13133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:37:55.700889 13133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:37:55.700977 13133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:37:55.705228 13133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:37:55.705328 13133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:37:55.705366 13133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:37:55.705425 13133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:37:55.705466 13133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:37:55.711417 13133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56413/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40057
--ranger_config_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:37:55.712653 13133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:37:55.713711 13133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:37:55.722443 13133 server_base.cc:1034] running on GCE node
W20260619 12:37:55.722404 13138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:55.722399 13139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:55.722675 13141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:55.723219 13133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:37:55.724459 13133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:37:55.725677 13133 hybrid_clock.cc:648] HybridClock initialized: now 1781872675725647 us; error 48 us; skew 500 ppm
Jun 19 12:37:55 dist-test-slave-kdrb krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:37:55.729049 13133 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:37:55.730158 13133 webserver.cc:460] Webserver started at http://127.0.98.62:41845/ using document root <none> and password file <none>
I20260619 12:37:55.730755 13133 fs_manager.cc:359] Metadata directory not provided
I20260619 12:37:55.730859 13133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 12:37:55 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:37:55.787976 13133 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:37:55.790381 13149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:55.791815 13133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:37:55.791957 13133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "45d023ad17dc4167969f3ba6eb3368a4"
format_stamp: "Formatted at 2026-06-19 12:37:53 on dist-test-slave-kdrb"
server_key: "a09d4c23a3b170538ad953648fb323ce"
server_key_iv: "e9bb6131db1cc00e052a8d9b22de0c64"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:55.792536 13133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:37:55.802351 13133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:37:55.805238 13133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:37:55.805444 13133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:37:55.813546 13133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40057
I20260619 12:37:55.813548 13201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40057 every 8 connection(s)
I20260619 12:37:55.814637 13133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:37:55.819046 13202 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:37:55.819317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 13133
I20260619 12:37:55.819861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43095
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40057
--webserver_port=42047
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56413/kms with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:37:55.821116 13202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4: Bootstrap starting.
I20260619 12:37:55.844889 13202 log.cc:824] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4: Log is configured to *not* fsync() on all Append() calls
I20260619 12:37:55.853709 13202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:37:55.854213 13202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4: Bootstrap complete.
I20260619 12:37:55.859707 13202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d023ad17dc4167969f3ba6eb3368a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } }
I20260619 12:37:55.860320 13202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d023ad17dc4167969f3ba6eb3368a4, State: Initialized, Role: FOLLOWER
I20260619 12:37:55.860898 13202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d023ad17dc4167969f3ba6eb3368a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } }
I20260619 12:37:55.861040 13202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:37:55.861133 13202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:37:55.861240 13202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:37:55.863899 13202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d023ad17dc4167969f3ba6eb3368a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } }
I20260619 12:37:55.864341 13202 leader_election.cc:302] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45d023ad17dc4167969f3ba6eb3368a4; no voters:
I20260619 12:37:55.864768 13202 leader_election.cc:288] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 12:37:55.864892 13206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 12:37:55.865185 13206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [term 2 LEADER]: Becoming Leader. State: Replica: 45d023ad17dc4167969f3ba6eb3368a4, State: Running, Role: LEADER
I20260619 12:37:55.865671 13206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d023ad17dc4167969f3ba6eb3368a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } }
I20260619 12:37:55.866086 13202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:37:55.867252 13208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45d023ad17dc4167969f3ba6eb3368a4. Latest consensus state: current_term: 2 leader_uuid: "45d023ad17dc4167969f3ba6eb3368a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d023ad17dc4167969f3ba6eb3368a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } } }
I20260619 12:37:55.867449 13208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [sys.catalog]: This master's current role is: LEADER
I20260619 12:37:55.867689 13207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "45d023ad17dc4167969f3ba6eb3368a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d023ad17dc4167969f3ba6eb3368a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } } }
I20260619 12:37:55.867777 13207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4 [sys.catalog]: This master's current role is: LEADER
I20260619 12:37:55.867901 13215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:37:55.868439 13202 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 12:37:55.874689 13215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:37:55.875746 13215 catalog_manager.cc:1259] Loaded cluster ID: c18ecf2673de43cebaa6d7ec5c5fc50e
I20260619 12:37:55.875818 13215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:37:55.879360 13215 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:37:55.880214 13215 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4: Loaded TSK: 0
I20260619 12:37:55.881551 13215 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:37:55.963438 13204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:37:55.963958 13204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:37:55.964056 13204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:37:55.971210 13204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:37:55.971308 13204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:37:55.971405 13204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:37:55.980468 13204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42047
--tserver_master_addrs=127.0.98.62:40057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:37:55.981910 13204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:37:55.983245 13204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:37:55.997062 13240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:55.997129 13239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:55.997207 13242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:55.997748 13204 server_base.cc:1034] running on GCE node
I20260619 12:37:55.998256 13204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:37:55.999114 13204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:37:56.000311 13204 hybrid_clock.cc:648] HybridClock initialized: now 1781872676000284 us; error 47 us; skew 500 ppm
Jun 19 12:37:55 dist-test-slave-kdrb krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:37:56.004029 13204 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:37:56.005244 13204 webserver.cc:460] Webserver started at http://127.0.98.1:42047/ using document root <none> and password file <none>
I20260619 12:37:56.005954 13204 fs_manager.cc:359] Metadata directory not provided
I20260619 12:37:56.006062 13204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 12:37:56 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:37:56.046957 13204 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 12:37:56.056403 13250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:56.058202 13204 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260619 12:37:56.058346 13204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "c0280ae0fd564958ab77e7fc26b9e972"
format_stamp: "Formatted at 2026-06-19 12:37:54 on dist-test-slave-kdrb"
server_key: "4c79e727e9a0d4d4d749e53c0480530f"
server_key_iv: "102874b611ab5f6311f5c8f867947cfe"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:56.059157 13204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:37:56.078394 13204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:37:56.082396 13204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:37:56.082630 13204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:37:56.083379 13204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:37:56.084497 13204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:37:56.084575 13204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:56.084645 13204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:37:56.084672 13204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:56.095788 13204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43095
I20260619 12:37:56.097177 13204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:37:56.098420 13363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43095 every 8 connection(s)
I20260619 12:37:56.101210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 13204
I20260619 12:37:56.101723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34795
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40057
--webserver_port=35023
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56413/kms with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:37:56 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:37:56.122551 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:56.099818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53049 (local address 127.0.98.62:40057)
0619 12:37:56.100494 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:56.100517 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:56.100976 (+ 459us) server_negotiation.cc:407] Connection header received
0619 12:37:56.103186 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:56.103210 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:56.103805 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:56.104214 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:37:56.106542 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:56.108539 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:56.110748 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:56.111754 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:56.111955 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:56.115717 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:37:56.115726 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:37:56.115747 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:37:56.115789 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:37:56.118132 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:56.118897 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:56.118905 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:56.118914 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:56.119085 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:56.119576 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:56.119580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:56.119582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:56.120128 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:37:56.120255 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:56.121067 (+ 812us) server_negotiation.cc:299] Negotiation successful
0619 12:37:56.121420 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":131,"threads_started":1}
I20260619 12:37:56.124897 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:56.099984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40057 (local address 127.0.98.1:53049)
0619 12:37:56.100786 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:56.100843 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:56.102544 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:56.104948 (+ 2404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:56.104960 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:56.105406 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:37:56.106209 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:56.106238 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:56.108778 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:56.108782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:56.110546 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:56.110565 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:56.112359 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:56.112368 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:56.112653 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:56.113547 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:37:56.113610 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:37:56.115384 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:37:56.118331 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:56.118338 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:56.118356 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:56.118741 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:56.119218 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:56.119221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:56.119223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:56.119429 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:56.120340 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:37:56.120352 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:37:56.120699 (+ 347us) client_negotiation.cc:769] Sending connection context
0619 12:37:56.123457 (+ 2758us) client_negotiation.cc:240] Negotiation successful
0619 12:37:56.123871 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":138,"threads_started":1}
I20260619 12:37:56.128367 13364 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40057
I20260619 12:37:56.128633 13364 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:56.129294 13364 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
W20260619 12:37:56.131776 13364 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40057 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 12:37:56.131898 13364 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:56.132077 13364 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
I20260619 12:37:56.133091 13364 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:56.133271 13364 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
W20260619 12:37:56.134064 13364 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:37:56.266515 13368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:37:56.267023 13368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:37:56.267124 13368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:37:56.273810 13368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:37:56.273903 13368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:37:56.273995 13368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:37:56.282359 13368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35023
--tserver_master_addrs=127.0.98.62:40057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:37:56.283843 13368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:37:56.285212 13368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:37:56.301200 13373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:56.301801 13376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:56.301211 13374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:56.303628 13368 server_base.cc:1034] running on GCE node
I20260619 12:37:56.304275 13368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:37:56.305285 13368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:37:56.308724 13368 hybrid_clock.cc:648] HybridClock initialized: now 1781872676308684 us; error 67 us; skew 500 ppm
Jun 19 12:37:56 dist-test-slave-kdrb krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:37:56.312598 13368 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:37:56.313947 13368 webserver.cc:460] Webserver started at http://127.0.98.2:35023/ using document root <none> and password file <none>
I20260619 12:37:56.314849 13368 fs_manager.cc:359] Metadata directory not provided
I20260619 12:37:56.314970 13368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 12:37:56 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:37:56.340646 13368 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:37:56.344159 13384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:56.346079 13368 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:37:56.346514 13368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "2a16a85af5504fe5ab553f2c5ef74e0d"
format_stamp: "Formatted at 2026-06-19 12:37:55 on dist-test-slave-kdrb"
server_key: "9c55eec14a4a64255a7d81e0e762f57d"
server_key_iv: "139201c54a0c1f34d1b5fb434028e6cc"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:56.347452 13368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:37:56.369508 13368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:37:56.374058 13368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:37:56.374377 13368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:37:56.375244 13368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:37:56.376459 13368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:37:56.376570 13368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:56.376684 13368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:37:56.376765 13368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:56.389390 13368 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34795
I20260619 12:37:56.391099 13368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:37:56.394235 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 13368
I20260619 12:37:56.394694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42051
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40057
--webserver_port=45229
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56413/kms with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:37:56.398667 13497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34795 every 8 connection(s)
Jun 19 12:37:56 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:37:56.416209 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:56.402552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50857 (local address 127.0.98.62:40057)
0619 12:37:56.402682 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:56.402687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:56.403139 (+ 452us) server_negotiation.cc:407] Connection header received
0619 12:37:56.404369 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:56.404373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:56.404426 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:56.404477 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:37:56.406051 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:56.406829 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:56.408531 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:56.408860 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:56.408930 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:56.412034 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:37:56.412038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:37:56.412040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:37:56.412072 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:37:56.413790 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:56.414408 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:56.414412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:56.414414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:56.414499 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:56.414875 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:56.414879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:56.414880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:56.415096 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:37:56.415171 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:56.415895 (+ 724us) server_negotiation.cc:299] Negotiation successful
0619 12:37:56.416032 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:37:56.418164 13501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:56.401443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40057 (local address 127.0.98.2:50857)
0619 12:37:56.402973 (+ 1530us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:56.403029 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:56.404106 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:56.404880 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:56.404889 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:56.405243 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:37:56.405882 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:56.405901 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:56.406990 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:56.406993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:56.408392 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:56.408403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:56.409127 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:56.409131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:56.409333 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:56.410074 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:37:56.410094 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:37:56.411835 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:37:56.413923 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:56.413929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:56.413946 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:56.414295 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:56.414599 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:56.414604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:56.414606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:56.414771 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:56.415247 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:37:56.415253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:37:56.415618 (+ 365us) client_negotiation.cc:769] Sending connection context
0619 12:37:56.416723 (+ 1105us) client_negotiation.cc:240] Negotiation successful
0619 12:37:56.417075 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1316,"thread_start_us":147,"threads_started":1}
I20260619 12:37:56.419857 13498 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40057
I20260619 12:37:56.420123 13498 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:56.420794 13498 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
W20260619 12:37:56.422616 13498 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40057 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 12:37:56.422750 13498 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:56.422938 13498 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
I20260619 12:37:56.423908 13498 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:56.424086 13498 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
W20260619 12:37:56.424885 13498 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:37:56.591579 13500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:37:56.592012 13500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:37:56.592111 13500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:37:56.597756 13500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:37:56.597864 13500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:37:56.597956 13500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:37:56.605839 13500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45229
--tserver_master_addrs=127.0.98.62:40057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:37:56.607335 13500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:37:56.608752 13500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:37:56.619757 13510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:37:56.619796 13507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:56.620051 13500 server_base.cc:1034] running on GCE node
W20260619 12:37:56.619797 13508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:37:56.622077 13500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:37:56.622923 13500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:37:56.626251 13500 hybrid_clock.cc:648] HybridClock initialized: now 1781872676626202 us; error 67 us; skew 500 ppm
Jun 19 12:37:56 dist-test-slave-kdrb krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:37:56.629493 13500 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:37:56.630714 13500 webserver.cc:460] Webserver started at http://127.0.98.3:45229/ using document root <none> and password file <none>
I20260619 12:37:56.631356 13500 fs_manager.cc:359] Metadata directory not provided
I20260619 12:37:56.631466 13500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 12:37:56 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:37:56.654996 13500 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:37:56.659667 13518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:56.661629 13500 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260619 12:37:56.661811 13500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "5b4e25a32ce745d894d14876c0478f16"
format_stamp: "Formatted at 2026-06-19 12:37:55 on dist-test-slave-kdrb"
server_key: "4deda2bec5d1d94e93d481c2b468680e"
server_key_iv: "89007633a847185b7f3619ee0a09b113"
server_key_version: "kuduclusterkey@0"
I20260619 12:37:56.662614 13500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:37:56.679225 13500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:37:56.683130 13500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:37:56.683348 13500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:37:56.684170 13500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:37:56.685273 13500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:37:56.685341 13500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:56.685400 13500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:37:56.685436 13500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:37:56.697613 13500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42051
I20260619 12:37:56.697781 13631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42051 every 8 connection(s)
I20260619 12:37:56.699035 13500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:37:56.708962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 13500
Jun 19 12:37:56 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:37:56.717381 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:56.701784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40057 (local address 127.0.98.3:36721)
0619 12:37:56.702427 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:56.702504 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:56.703719 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:56.704382 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:56.704392 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:56.704799 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:37:56.705544 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:56.705566 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:56.706735 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:56.706738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:56.707745 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:56.707754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:56.708383 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:56.708386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:56.708543 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:56.709300 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:37:56.709322 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:37:56.711341 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:37:56.713702 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:56.713710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:56.713731 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:56.714146 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:56.714492 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:37:56.714496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:37:56.714498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:37:56.714715 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:37:56.715241 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:37:56.715248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:37:56.715665 (+ 417us) client_negotiation.cc:769] Sending connection context
0619 12:37:56.715879 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 12:37:56.716267 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":163,"threads_started":1}
I20260619 12:37:56.718291 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:56.701240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36721 (local address 127.0.98.62:40057)
0619 12:37:56.701402 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:56.701406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:56.702643 (+ 1237us) server_negotiation.cc:407] Connection header received
0619 12:37:56.703988 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:56.703992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:56.704058 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:56.704170 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:37:56.705733 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:56.706593 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:56.707867 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:56.708218 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:56.708343 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:56.711607 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:37:56.711612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:37:56.711615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:37:56.711647 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:37:56.713540 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:56.714284 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:56.714287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:56.714289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:56.714385 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:56.714844 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:56.714847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:56.714849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:56.715093 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:37:56.715183 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:56.717989 (+ 2806us) server_negotiation.cc:299] Negotiation successful
0619 12:37:56.718136 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:37:56.719277 13632 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40057
I20260619 12:37:56.719587 13632 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:56.720249 13632 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
W20260619 12:37:56.722231 13632 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40057 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 12:37:56.722383 13632 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:56.722612 13632 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
I20260619 12:37:56.723639 13632 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:56.723847 13632 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
W20260619 12:37:56.724594 13632 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 12:37:56.730507 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:56.710827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.0.98.62:40057)
0619 12:37:56.711695 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:56.711701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:56.711717 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:37:56.711757 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:56.711760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:56.711809 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:56.711904 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:37:56.717930 (+ 6026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:56.718711 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:56.721748 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:56.723058 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:56.723204 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:56.726587 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:37:56.726590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:37:56.726592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:37:56.726638 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:37:56.728584 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:56.729158 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:56.729161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:56.729163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:56.729264 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:56.729579 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:56.729581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:56.729582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:56.729794 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:37:56.729944 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:56.730172 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 12:37:56.730294 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":93,"threads_started":1}
I20260619 12:37:57.134310 13364 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:57.134647 13364 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
I20260619 12:37:57.136106 13166 ts_manager.cc:194] Registered new tserver with Master: c0280ae0fd564958ab77e7fc26b9e972 (127.0.98.1:43095)
I20260619 12:37:57.137930 13166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53049
Jun 19 12:37:57 dist-test-slave-kdrb krb5kdc[12170](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781872677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:37:57.425133 13498 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:57.425493 13498 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
I20260619 12:37:57.426362 13166 ts_manager.cc:194] Registered new tserver with Master: 2a16a85af5504fe5ab553f2c5ef74e0d (127.0.98.2:34795)
I20260619 12:37:57.426975 13166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50857
Jun 19 12:37:57 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:37:57.724864 13632 heartbeater.cc:464] Registering TS with master...
I20260619 12:37:57.725175 13632 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
I20260619 12:37:57.726042 13163 ts_manager.cc:194] Registered new tserver with Master: 5b4e25a32ce745d894d14876c0478f16 (127.0.98.3:42051)
I20260619 12:37:57.726879 13163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36721
I20260619 12:37:57.734264 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:37:57.749516 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:57.735973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.0.98.62:40057)
0619 12:37:57.739735 (+ 3762us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:57.739740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:57.739757 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:37:57.739803 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:57.739806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:57.739862 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:57.739971 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:37:57.741052 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.741940 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.743173 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.743560 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.743679 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:57.745379 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:37:57.745383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:37:57.745386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:37:57.745416 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:37:57.747275 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:57.747797 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:57.747800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:57.747802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:57.747893 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:37:57.748216 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:37:57.748219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:37:57.748221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:37:57.748452 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:37:57.748574 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:57.749202 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 12:37:57.749339 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3663,"thread_start_us":95,"threads_started":1}
I20260619 12:37:57.754484 13163 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:37:57.759148 13163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:37:57.780448 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:57.771642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42051 (local address 127.0.0.1:41784)
0619 12:37:57.774211 (+ 2569us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:57.774225 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:57.774329 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:57.774847 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:57.774853 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:57.774878 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:37:57.775017 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:57.775024 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.776650 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.776654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:57.778081 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:57.778095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.779834 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.779838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:57.779976 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:57.780020 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 12:37:57.780150 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 12:37:57.780285 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2511,"thread_start_us":109,"threads_started":1}
I20260619 12:37:57.780465 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:57.771796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.0.98.3:42051)
0619 12:37:57.774081 (+ 2285us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:57.774088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:57.774341 (+ 253us) server_negotiation.cc:407] Connection header received
0619 12:37:57.774437 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:57.774443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:57.774671 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:57.774839 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:37:57.775142 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.776488 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.778250 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.779701 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.779828 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:57.779991 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:57.780201 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 12:37:57.780307 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2176,"thread_start_us":54,"threads_started":1}
I20260619 12:37:57.784008 13549 tablet_service.cc:1467] Processing CreateTablet for tablet c8ef90de38bd461ea913884b832a5906 (DEFAULT_TABLE table=test-table [id=4b76bc37d5cf40b688d256376380db10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:37:57.785236 13549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ef90de38bd461ea913884b832a5906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:37:57.786078 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:57.773074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43095 (local address 127.0.0.1:58806)
0619 12:37:57.780999 (+ 7925us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:57.781013 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:57.781093 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:57.781777 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:57.781779 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:57.781795 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:37:57.781867 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:57.781872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.783266 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.783269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:57.784375 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:57.784385 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.785662 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.785666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:57.785752 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:57.785776 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 12:37:57.785850 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 12:37:57.785921 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7867,"thread_start_us":89,"threads_started":1}
I20260619 12:37:57.786473 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:57.771563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34795 (local address 127.0.0.1:57286)
0619 12:37:57.773601 (+ 2038us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:57.773655 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:57.773796 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:57.779862 (+ 6066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:57.779865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:57.779885 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:37:57.779985 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:57.779991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.781627 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.781630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:57.782695 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:57.782705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.786092 (+ 3387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.786096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:57.786189 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:57.786217 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 12:37:57.786267 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 12:37:57.786337 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1793,"thread_start_us":120,"threads_started":1}
I20260619 12:37:57.786506 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:57.773241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58806 (local address 127.0.98.1:43095)
0619 12:37:57.773543 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:57.773551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:57.781195 (+ 7644us) server_negotiation.cc:407] Connection header received
0619 12:37:57.781280 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:57.781286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:57.781491 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:57.781626 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:37:57.781985 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.783132 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.784517 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.785352 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.785457 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:57.785596 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:57.786270 (+ 674us) server_negotiation.cc:299] Negotiation successful
0619 12:37:57.786373 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
I20260619 12:37:57.788196 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:57.772439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.0.98.2:34795)
0619 12:37:57.779313 (+ 6874us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:57.779321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:57.779347 (+ 26us) server_negotiation.cc:407] Connection header received
0619 12:37:57.779432 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:57.779437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:57.779703 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:57.780180 (+ 477us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:37:57.780335 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.781394 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.782850 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.783655 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.787700 (+ 4045us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:57.787878 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:57.787934 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 12:37:57.788038 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6751,"thread_start_us":125,"threads_started":1}
I20260619 12:37:57.789721 13298 tablet_service.cc:1467] Processing CreateTablet for tablet c8ef90de38bd461ea913884b832a5906 (DEFAULT_TABLE table=test-table [id=4b76bc37d5cf40b688d256376380db10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:37:57.790867 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ef90de38bd461ea913884b832a5906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:37:57.791221 13432 tablet_service.cc:1467] Processing CreateTablet for tablet c8ef90de38bd461ea913884b832a5906 (DEFAULT_TABLE table=test-table [id=4b76bc37d5cf40b688d256376380db10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:37:57.792321 13432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ef90de38bd461ea913884b832a5906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:37:57.799686 13665 tablet_bootstrap.cc:492] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16: Bootstrap starting.
I20260619 12:37:57.802207 13665 tablet_bootstrap.cc:654] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16: Neither blocks nor log segments found. Creating new log.
I20260619 12:37:57.803213 13665 log.cc:824] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16: Log is configured to *not* fsync() on all Append() calls
I20260619 12:37:57.811575 13668 tablet_bootstrap.cc:492] T c8ef90de38bd461ea913884b832a5906 P c0280ae0fd564958ab77e7fc26b9e972: Bootstrap starting.
I20260619 12:37:57.816545 13665 tablet_bootstrap.cc:492] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16: No bootstrap required, opened a new log
I20260619 12:37:57.816934 13665 ts_tablet_manager.cc:1397] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260619 12:37:57.818130 13667 tablet_bootstrap.cc:492] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d: Bootstrap starting.
I20260619 12:37:57.817521 13668 tablet_bootstrap.cc:654] T c8ef90de38bd461ea913884b832a5906 P c0280ae0fd564958ab77e7fc26b9e972: Neither blocks nor log segments found. Creating new log.
I20260619 12:37:57.820564 13668 log.cc:824] T c8ef90de38bd461ea913884b832a5906 P c0280ae0fd564958ab77e7fc26b9e972: Log is configured to *not* fsync() on all Append() calls
I20260619 12:37:57.821907 13665 raft_consensus.cc:348] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a16a85af5504fe5ab553f2c5ef74e0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "5b4e25a32ce745d894d14876c0478f16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42051 } } peers { permanent_uuid: "c0280ae0fd564958ab77e7fc26b9e972" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43095 } }
I20260619 12:37:57.822373 13665 raft_consensus.cc:374] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:37:57.822379 13667 tablet_bootstrap.cc:654] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d: Neither blocks nor log segments found. Creating new log.
I20260619 12:37:57.822499 13665 raft_consensus.cc:729] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b4e25a32ce745d894d14876c0478f16, State: Initialized, Role: FOLLOWER
I20260619 12:37:57.823132 13665 consensus_queue.cc:260] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a16a85af5504fe5ab553f2c5ef74e0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "5b4e25a32ce745d894d14876c0478f16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42051 } } peers { permanent_uuid: "c0280ae0fd564958ab77e7fc26b9e972" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43095 } }
I20260619 12:37:57.824508 13665 ts_tablet_manager.cc:1428] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 12:37:57.824743 13667 log.cc:824] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d: Log is configured to *not* fsync() on all Append() calls
I20260619 12:37:57.824941 13632 heartbeater.cc:503] Master 127.0.98.62:40057 was elected leader, sending a full tablet report...
I20260619 12:37:57.827862 13668 tablet_bootstrap.cc:492] T c8ef90de38bd461ea913884b832a5906 P c0280ae0fd564958ab77e7fc26b9e972: No bootstrap required, opened a new log
I20260619 12:37:57.828137 13668 ts_tablet_manager.cc:1397] T c8ef90de38bd461ea913884b832a5906 P c0280ae0fd564958ab77e7fc26b9e972: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.002s
I20260619 12:37:57.830423 13667 tablet_bootstrap.cc:492] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d: No bootstrap required, opened a new log
I20260619 12:37:57.830709 13667 ts_tablet_manager.cc:1397] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260619 12:37:57.832499 13668 raft_consensus.cc:348] T c8ef90de38bd461ea913884b832a5906 P c0280ae0fd564958ab77e7fc26b9e972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a16a85af5504fe5ab553f2c5ef74e0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "5b4e25a32ce745d894d14876c0478f16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42051 } } peers { permanent_uuid: "c0280ae0fd564958ab77e7fc26b9e972" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43095 } }
I20260619 12:37:57.832877 13668 raft_consensus.cc:374] T c8ef90de38bd461ea913884b832a5906 P c0280ae0fd564958ab77e7fc26b9e972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:37:57.832985 13668 raft_consensus.cc:729] T c8ef90de38bd461ea913884b832a5906 P c0280ae0fd564958ab77e7fc26b9e972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0280ae0fd564958ab77e7fc26b9e972, State: Initialized, Role: FOLLOWER
I20260619 12:37:57.833559 13668 consensus_queue.cc:260] T c8ef90de38bd461ea913884b832a5906 P c0280ae0fd564958ab77e7fc26b9e972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a16a85af5504fe5ab553f2c5ef74e0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "5b4e25a32ce745d894d14876c0478f16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42051 } } peers { permanent_uuid: "c0280ae0fd564958ab77e7fc26b9e972" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43095 } }
I20260619 12:37:57.834798 13668 ts_tablet_manager.cc:1428] T c8ef90de38bd461ea913884b832a5906 P c0280ae0fd564958ab77e7fc26b9e972: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260619 12:37:57.835201 13364 heartbeater.cc:503] Master 127.0.98.62:40057 was elected leader, sending a full tablet report...
I20260619 12:37:57.835299 13667 raft_consensus.cc:348] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a16a85af5504fe5ab553f2c5ef74e0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "5b4e25a32ce745d894d14876c0478f16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42051 } } peers { permanent_uuid: "c0280ae0fd564958ab77e7fc26b9e972" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43095 } }
I20260619 12:37:57.835670 13667 raft_consensus.cc:374] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:37:57.835737 13667 raft_consensus.cc:729] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a16a85af5504fe5ab553f2c5ef74e0d, State: Initialized, Role: FOLLOWER
I20260619 12:37:57.836218 13667 consensus_queue.cc:260] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a16a85af5504fe5ab553f2c5ef74e0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "5b4e25a32ce745d894d14876c0478f16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42051 } } peers { permanent_uuid: "c0280ae0fd564958ab77e7fc26b9e972" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43095 } }
I20260619 12:37:57.837337 13667 ts_tablet_manager.cc:1428] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 12:37:57.837741 13498 heartbeater.cc:503] Master 127.0.98.62:40057 was elected leader, sending a full tablet report...
W20260619 12:37:57.853472 13365 tablet.cc:2369] T c8ef90de38bd461ea913884b832a5906 P c0280ae0fd564958ab77e7fc26b9e972: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 12:37:57 dist-test-slave-kdrb krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 12:37:57.901320 13499 tablet.cc:2369] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:37:57.918167 13670 raft_consensus.cc:482] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:37:57.918418 13670 raft_consensus.cc:504] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a16a85af5504fe5ab553f2c5ef74e0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "5b4e25a32ce745d894d14876c0478f16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42051 } } peers { permanent_uuid: "c0280ae0fd564958ab77e7fc26b9e972" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43095 } }
I20260619 12:37:57.919819 13670 leader_election.cc:288] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a16a85af5504fe5ab553f2c5ef74e0d (127.0.98.2:34795), c0280ae0fd564958ab77e7fc26b9e972 (127.0.98.1:43095)
I20260619 12:37:57.926098 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:57.920374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48949 (local address 127.0.98.1:43095)
0619 12:37:57.920515 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:57.920520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:57.920710 (+ 190us) server_negotiation.cc:407] Connection header received
0619 12:37:57.920899 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:57.920903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:57.920955 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:57.921003 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:37:57.922238 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.922962 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.924758 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.925360 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.925450 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:57.925528 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:57.925876 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 12:37:57.925957 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:37:57.926946 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:57.920291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43095 (local address 127.0.98.3:48949)
0619 12:37:57.920617 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:57.920629 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:57.920765 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:57.921437 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:57.921440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:57.921462 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:37:57.921605 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:57.921612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.923301 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.923304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:57.924609 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:57.924618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.925616 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.925619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:57.925739 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:57.925762 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:37:57.926655 (+ 893us) client_negotiation.cc:240] Negotiation successful
0619 12:37:57.926774 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":134,"threads_started":1}
I20260619 12:37:57.927299 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:57.921392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48373 (local address 127.0.98.2:34795)
0619 12:37:57.921511 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:57.921515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:57.921530 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:37:57.921982 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:57.921986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:57.922040 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:57.922090 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:37:57.923107 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.923921 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.925492 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.926104 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.926196 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:57.926257 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:57.927077 (+ 820us) server_negotiation.cc:299] Negotiation successful
0619 12:37:57.927188 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 12:37:57.927642 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ef90de38bd461ea913884b832a5906" candidate_uuid: "5b4e25a32ce745d894d14876c0478f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0280ae0fd564958ab77e7fc26b9e972" is_pre_election: true
I20260619 12:37:57.927894 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:57.921015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34795 (local address 127.0.98.3:48373)
0619 12:37:57.921293 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:57.921307 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:57.921733 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:57.922381 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:37:57.922384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:37:57.922402 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:37:57.922962 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:57.922972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.924228 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.924231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:57.925355 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:37:57.925364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.926822 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.926825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:37:57.926925 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:57.926950 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 12:37:57.927445 (+ 495us) client_negotiation.cc:240] Negotiation successful
0619 12:37:57.927769 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":110,"threads_started":1}
I20260619 12:37:57.928037 13318 raft_consensus.cc:2454] T c8ef90de38bd461ea913884b832a5906 P c0280ae0fd564958ab77e7fc26b9e972 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b4e25a32ce745d894d14876c0478f16 in term 0.
I20260619 12:37:57.928524 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ef90de38bd461ea913884b832a5906" candidate_uuid: "5b4e25a32ce745d894d14876c0478f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a16a85af5504fe5ab553f2c5ef74e0d" is_pre_election: true
I20260619 12:37:57.928563 13521 leader_election.cc:302] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b4e25a32ce745d894d14876c0478f16, c0280ae0fd564958ab77e7fc26b9e972; no voters:
I20260619 12:37:57.928797 13444 raft_consensus.cc:2454] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b4e25a32ce745d894d14876c0478f16 in term 0.
I20260619 12:37:57.928903 13670 raft_consensus.cc:2781] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:37:57.928999 13670 raft_consensus.cc:482] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:37:57.929050 13670 raft_consensus.cc:3037] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:37:57.930641 13670 raft_consensus.cc:504] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a16a85af5504fe5ab553f2c5ef74e0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "5b4e25a32ce745d894d14876c0478f16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42051 } } peers { permanent_uuid: "c0280ae0fd564958ab77e7fc26b9e972" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43095 } }
I20260619 12:37:57.931016 13670 leader_election.cc:288] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [CANDIDATE]: Term 1 election: Requested vote from peers 2a16a85af5504fe5ab553f2c5ef74e0d (127.0.98.2:34795), c0280ae0fd564958ab77e7fc26b9e972 (127.0.98.1:43095)
I20260619 12:37:57.931357 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ef90de38bd461ea913884b832a5906" candidate_uuid: "5b4e25a32ce745d894d14876c0478f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a16a85af5504fe5ab553f2c5ef74e0d"
I20260619 12:37:57.931516 13444 raft_consensus.cc:3037] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:37:57.931587 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ef90de38bd461ea913884b832a5906" candidate_uuid: "5b4e25a32ce745d894d14876c0478f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0280ae0fd564958ab77e7fc26b9e972"
I20260619 12:37:57.931708 13318 raft_consensus.cc:3037] T c8ef90de38bd461ea913884b832a5906 P c0280ae0fd564958ab77e7fc26b9e972 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:37:57.932680 13444 raft_consensus.cc:2454] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b4e25a32ce745d894d14876c0478f16 in term 1.
I20260619 12:37:57.932963 13318 raft_consensus.cc:2454] T c8ef90de38bd461ea913884b832a5906 P c0280ae0fd564958ab77e7fc26b9e972 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b4e25a32ce745d894d14876c0478f16 in term 1.
I20260619 12:37:57.933017 13522 leader_election.cc:302] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a16a85af5504fe5ab553f2c5ef74e0d, 5b4e25a32ce745d894d14876c0478f16; no voters:
I20260619 12:37:57.933382 13670 raft_consensus.cc:2781] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:37:57.933630 13670 raft_consensus.cc:686] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [term 1 LEADER]: Becoming Leader. State: Replica: 5b4e25a32ce745d894d14876c0478f16, State: Running, Role: LEADER
I20260619 12:37:57.934013 13670 consensus_queue.cc:237] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a16a85af5504fe5ab553f2c5ef74e0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "5b4e25a32ce745d894d14876c0478f16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42051 } } peers { permanent_uuid: "c0280ae0fd564958ab77e7fc26b9e972" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43095 } }
I20260619 12:37:57.938516 13163 catalog_manager.cc:5495] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b4e25a32ce745d894d14876c0478f16 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5b4e25a32ce745d894d14876c0478f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a16a85af5504fe5ab553f2c5ef74e0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b4e25a32ce745d894d14876c0478f16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0280ae0fd564958ab77e7fc26b9e972" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43095 } health_report { overall_health: UNKNOWN } } }
W20260619 12:37:57.951285 13633 tablet.cc:2369] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:37:57.976799 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:57.971628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.0.98.3:42051)
0619 12:37:57.971753 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 12:37:57.971758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:37:57.971775 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:37:57.971988 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:37:57.971992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:37:57.972065 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:37:57.972188 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:37:57.972633 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.973433 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.974682 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:37:57.974921 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:37:57.975040 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:37:57.975372 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:37:57.975988 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:37:57.976115 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 12:37:57.976552 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 12:37:57.976616 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:37:57.984560 13318 raft_consensus.cc:1261] T c8ef90de38bd461ea913884b832a5906 P c0280ae0fd564958ab77e7fc26b9e972 [term 1 FOLLOWER]: Refusing update from remote peer 5b4e25a32ce745d894d14876c0478f16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:37:57.985412 13670 consensus_queue.cc:1035] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0280ae0fd564958ab77e7fc26b9e972" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:37:57.987156 13444 raft_consensus.cc:1261] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d [term 1 FOLLOWER]: Refusing update from remote peer 5b4e25a32ce745d894d14876c0478f16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:37:57.987826 13670 consensus_queue.cc:1035] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a16a85af5504fe5ab553f2c5ef74e0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:37:57.997658 13681 mvcc.cc:205] Tried to move back new op lower bound from 7298550489015115776 to 7298550488823144448. Current Snapshot: MvccSnapshot[applied={T|T < 7298550489015115776}]
I20260619 12:37:58.000077 13682 mvcc.cc:205] Tried to move back new op lower bound from 7298550489015115776 to 7298550488823144448. Current Snapshot: MvccSnapshot[applied={T|T < 7298550489015115776}]
I20260619 12:37:58.005964 13684 mvcc.cc:205] Tried to move back new op lower bound from 7298550489015115776 to 7298550488823144448. Current Snapshot: MvccSnapshot[applied={T|T < 7298550489015115776}]
I20260619 12:37:58.017995 13163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:37:58.018210 13163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:37:58.021294 13163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45d023ad17dc4167969f3ba6eb3368a4: Sending DeleteTablet for 3 replicas of tablet c8ef90de38bd461ea913884b832a5906
I20260619 12:37:58.022643 13432 tablet_service.cc:1514] Processing DeleteTablet for tablet c8ef90de38bd461ea913884b832a5906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:37:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57286
I20260619 12:37:58.023214 13298 tablet_service.cc:1514] Processing DeleteTablet for tablet c8ef90de38bd461ea913884b832a5906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:37:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58806
I20260619 12:37:58.024225 13549 tablet_service.cc:1514] Processing DeleteTablet for tablet c8ef90de38bd461ea913884b832a5906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:37:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41784
I20260619 12:37:58.025471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 13204
I20260619 12:37:58.032972 13691 tablet_replica.cc:330] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d: stopping tablet replica
I20260619 12:37:58.033485 13691 raft_consensus.cc:2229] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:37:58.033972 13691 raft_consensus.cc:2258] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:37:58.035837 13691 ts_tablet_manager.cc:1905] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:37:58.036369 13693 tablet_replica.cc:330] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16: stopping tablet replica
I20260619 12:37:58.036772 13693 raft_consensus.cc:2229] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:37:58.037317 13693 raft_consensus.cc:2258] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:37:58.039398 13691 ts_tablet_manager.cc:1918] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:37:58.039587 13691 log.cc:1196] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/c8ef90de38bd461ea913884b832a5906
I20260619 12:37:58.039978 13691 ts_tablet_manager.cc:1939] T c8ef90de38bd461ea913884b832a5906 P 2a16a85af5504fe5ab553f2c5ef74e0d: Deleting consensus metadata
I20260619 12:37:58.040875 13693 ts_tablet_manager.cc:1905] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:37:58.041534 13153 catalog_manager.cc:4841] TS 2a16a85af5504fe5ab553f2c5ef74e0d (127.0.98.2:34795): tablet c8ef90de38bd461ea913884b832a5906 (table test-table [id=4b76bc37d5cf40b688d256376380db10]) successfully deleted
I20260619 12:37:58.043413 13693 ts_tablet_manager.cc:1918] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260619 12:37:58.043944 13152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43095 (error 108)
I20260619 12:37:58.045037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 13368
I20260619 12:37:58.047616 13693 log.cc:1196] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/c8ef90de38bd461ea913884b832a5906
I20260619 12:37:58.049057 13693 ts_tablet_manager.cc:1939] T c8ef90de38bd461ea913884b832a5906 P 5b4e25a32ce745d894d14876c0478f16: Deleting consensus metadata
I20260619 12:37:58.053249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 13500
I20260619 12:37:58.064810 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:37:58.064480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43095 (local address 127.0.0.1:58810)
0619 12:37:58.064608 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:58.064690 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 12:37:58.065078 13152 catalog_manager.cc:4584] TS c0280ae0fd564958ab77e7fc26b9e972 (127.0.98.1:43095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8ef90de38bd461ea913884b832a5906: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43095: connect: Connection refused (error 111)
W20260619 12:37:58.066322 13663 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:37:58.065881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42051 (local address 127.0.0.1:41798)
0619 12:37:58.065971 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 12:37:58.065987 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:37:58.066087 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:37:58.066191 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:42051: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 12:37:58.066402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 13133
I20260619 12:37:58.071949 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 12:37:58.407342 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 12:37:58.407505 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 12:37:58.744380 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 12:37:58.744 UTC [12200] LOG: received smart shutdown request
2026-06-19 12:37:58.748 UTC [12200] LOG: background worker "logical replication launcher" (PID 12209) exited with exit code 1
2026-06-19 12:37:58.748 UTC [12204] LOG: shutting down
2026-06-19 12:37:58.924 UTC [12200] LOG: database system is shut down
2026-06-19T12:37:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50221 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:37:59 dist-test-slave-kdrb krb5kdc[13703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:37:59 dist-test-slave-kdrb krb5kdc[13703](info): set up 2 sockets
Jun 19 12:37:59 dist-test-slave-kdrb krb5kdc[13703](info): commencing operation
krb5kdc: starting...
W20260619 12:38:00.083442 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:38:00 dist-test-slave-kdrb krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:38:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:38:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:38:00.211092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33093
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33093,127.0.98.61:44579,127.0.98.60:35447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:38:00.322078 13716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:00.322427 13716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:00.322518 13716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:00.326587 13716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:38:00.326689 13716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:00.326723 13716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:00.326746 13716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:38:00.326781 13716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:38:00.332154 13716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33093,127.0.98.61:44579,127.0.98.60:35447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:00.333346 13716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:00.334439 13716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:00.343608 13724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:00.343608 13721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:00.343760 13716 server_base.cc:1034] running on GCE node
W20260619 12:38:00.343946 13722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:00.344565 13716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:00.345712 13716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:00.346920 13716 hybrid_clock.cc:648] HybridClock initialized: now 1781872680346898 us; error 38 us; skew 500 ppm
Jun 19 12:38:00 dist-test-slave-kdrb krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:00.349831 13716 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:38:00.350754 13716 webserver.cc:460] Webserver started at http://127.0.98.62:35495/ using document root <none> and password file <none>
I20260619 12:38:00.351346 13716 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:00.351439 13716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:00.351696 13716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:38:00.353781 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "5099accf4e02456a92a9c1961b07c668"
format_stamp: "Formatted at 2026-06-19 12:38:00 on dist-test-slave-kdrb"
server_key: "fcbf04cf44c965d1b49c2b02b7927da9"
server_key_iv: "92ff77b89b55dacaf44cd7239f5802ba"
server_key_version: "encryptionkey@0"
I20260619 12:38:00.354364 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "5099accf4e02456a92a9c1961b07c668"
format_stamp: "Formatted at 2026-06-19 12:38:00 on dist-test-slave-kdrb"
server_key: "fcbf04cf44c965d1b49c2b02b7927da9"
server_key_iv: "92ff77b89b55dacaf44cd7239f5802ba"
server_key_version: "encryptionkey@0"
I20260619 12:38:00.358336 13716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 12:38:00.360941 13731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:00.362196 13716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:38:00.362349 13716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "5099accf4e02456a92a9c1961b07c668"
format_stamp: "Formatted at 2026-06-19 12:38:00 on dist-test-slave-kdrb"
server_key: "fcbf04cf44c965d1b49c2b02b7927da9"
server_key_iv: "92ff77b89b55dacaf44cd7239f5802ba"
server_key_version: "encryptionkey@0"
I20260619 12:38:00.362470 13716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:00.378082 13716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:00.381826 13716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:00.382053 13716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:00.389808 13716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33093
I20260619 12:38:00.389832 13783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33093 every 8 connection(s)
I20260619 12:38:00.390867 13716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:38:00.393733 13784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:00.397619 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:38:00.397907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 13716
I20260619 12:38:00.398032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:38:00.398313 392 external_mini_cluster.cc:1428] Setting key d6952ee56ee34ffb9eb601289db85783
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 12:38:00 dist-test-slave-kdrb krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:38:00.415851 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.401178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34066 (local address 127.0.98.62:33093)
0619 12:38:00.401532 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:00.401546 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:00.401596 (+ 50us) server_negotiation.cc:407] Connection header received
0619 12:38:00.402138 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:00.402146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:00.402426 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:00.402639 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:00.403465 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.405007 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.406058 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.406691 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.406896 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.410336 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:00.410342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:00.410354 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:00.410378 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:00.412462 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.412965 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.412971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.412973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.413086 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.413409 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.413415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.413417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.413849 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:00.413997 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:00.414391 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 12:38:00.414668 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":215,"thread_start_us":100,"threads_started":1}
I20260619 12:38:00.415848 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.400230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33093 (local address 127.0.0.1:34066)
0619 12:38:00.400741 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.400804 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:00.401784 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:00.402662 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:00.402666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:00.403014 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:00.403301 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.403316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.405168 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.405171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.405928 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.405937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.406855 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.406858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.407008 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.407674 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:00.407697 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:00.410124 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:00.412588 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.412598 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.412610 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.412847 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.413190 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.413193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.413195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.413317 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.413975 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:00.413981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:00.414130 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 12:38:00.414319 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 12:38:00.414722 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":158,"thread_start_us":146,"threads_started":1}
I20260619 12:38:00.418597 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:38:00.419457 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.419140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44579 (local address 127.0.0.1:34026)
0619 12:38:00.419296 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.419375 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 12:38:00.419823 13735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44579: connect: Connection refused (error 111)
I20260619 12:38:00.420771 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.420544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44579 (local address 127.0.0.1:34030)
0619 12:38:00.420657 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.420706 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 12:38:00.420943 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44579: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44579: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 12:38:00.426422 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44579
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33093,127.0.98.61:44579,127.0.98.60:35447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:38:00.470228 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } attempt: 1
I20260619 12:38:00.471256 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.470947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44579 (local address 127.0.0.1:34038)
0619 12:38:00.471112 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.471179 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 12:38:00.472081 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.471848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44579 (local address 127.0.0.1:34050)
0619 12:38:00.471977 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.472027 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 12:38:00.472327 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44579: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44579: connect: Connection refused (error 111)
W20260619 12:38:00.539768 13791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:00.540046 13791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:00.540089 13791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 12:38:00.540537 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } attempt: 2
I20260619 12:38:00.541435 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.541115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44579 (local address 127.0.0.1:34056)
0619 12:38:00.541283 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.541348 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 12:38:00.542235 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.542000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44579 (local address 127.0.0.1:34072)
0619 12:38:00.542122 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.542178 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 12:38:00.542394 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44579: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44579: connect: Connection refused (error 111)
W20260619 12:38:00.544773 13791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:38:00.544867 13791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:00.544934 13791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:00.544993 13791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:38:00.545032 13791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:38:00.552433 13791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33093,127.0.98.61:44579,127.0.98.60:35447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:00.553900 13791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:00.555017 13791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:00.563970 13797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:00.563977 13796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:00.563968 13799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:00.564344 13791 server_base.cc:1034] running on GCE node
I20260619 12:38:00.564906 13791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:00.566208 13791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:00.567389 13791 hybrid_clock.cc:648] HybridClock initialized: now 1781872680567339 us; error 65 us; skew 500 ppm
Jun 19 12:38:00 dist-test-slave-kdrb krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:00.570317 13791 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 12:38:00.571225 13791 webserver.cc:460] Webserver started at http://127.0.98.61:38593/ using document root <none> and password file <none>
I20260619 12:38:00.571832 13791 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:00.571897 13791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:00.572074 13791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:38:00.574112 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/data/instance:
uuid: "4d987fd1d88543ee99790d1da1cd10b0"
format_stamp: "Formatted at 2026-06-19 12:38:00 on dist-test-slave-kdrb"
server_key: "1574fd72831d22b8f447750c9cf9cb8d"
server_key_iv: "26df0d0333f6b0e7772d7329cd38a91a"
server_key_version: "encryptionkey@0"
I20260619 12:38:00.574653 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/wal/instance:
uuid: "4d987fd1d88543ee99790d1da1cd10b0"
format_stamp: "Formatted at 2026-06-19 12:38:00 on dist-test-slave-kdrb"
server_key: "1574fd72831d22b8f447750c9cf9cb8d"
server_key_iv: "26df0d0333f6b0e7772d7329cd38a91a"
server_key_version: "encryptionkey@0"
I20260619 12:38:00.578481 13791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 12:38:00.581032 13806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:00.582331 13791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:38:00.582475 13791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/wal
uuid: "4d987fd1d88543ee99790d1da1cd10b0"
format_stamp: "Formatted at 2026-06-19 12:38:00 on dist-test-slave-kdrb"
server_key: "1574fd72831d22b8f447750c9cf9cb8d"
server_key_iv: "26df0d0333f6b0e7772d7329cd38a91a"
server_key_version: "encryptionkey@0"
I20260619 12:38:00.582597 13791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:00.609812 13791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:00.613149 13791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:00.613374 13791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:00.620725 13791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44579
I20260619 12:38:00.620749 13858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44579 every 8 connection(s)
I20260619 12:38:00.621723 13791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/data/info.pb
I20260619 12:38:00.623317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 13791
I20260619 12:38:00.623446 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-1/wal/instance
I20260619 12:38:00.623800 392 external_mini_cluster.cc:1428] Setting key 3f5ed758a9370892de6d5f26b6d3e1a7
I20260619 12:38:00.625011 13859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:00.629081 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 12:38:00.633562 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 12:38:00 dist-test-slave-kdrb krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 12:38:00 dist-test-slave-kdrb krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 12:38:00.645318 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.631162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34076 (local address 127.0.98.62:33093)
0619 12:38:00.631300 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:00.631304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:00.632462 (+ 1158us) server_negotiation.cc:407] Connection header received
0619 12:38:00.633585 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:00.633589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:00.633664 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:00.633750 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:00.635451 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.636291 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.637684 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.637925 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.638045 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.641083 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:00.641086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:00.641088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:00.641111 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:00.642719 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.643361 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.643365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.643367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.643442 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.643924 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.643928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.643931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.644174 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:00.644259 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:00.644949 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 12:38:00.645113 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:38:00.646695 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.631727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33093 (local address 127.0.0.1:34076)
0619 12:38:00.632233 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.632297 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:00.633242 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:00.633924 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:00.633960 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:00.634512 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:00.635267 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.635290 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.636424 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.636430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.637558 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.637569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.638090 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.638094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.638304 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.638972 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:00.638996 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:00.640912 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:00.642837 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.642843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.642861 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.643251 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.643587 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.643592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.643597 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.643811 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.644298 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:00.644305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:00.644647 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 12:38:00.644881 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 12:38:00.645271 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":127,"threads_started":1}
I20260619 12:38:00.646775 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.634057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44579 (local address 127.0.0.1:34080)
0619 12:38:00.634166 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.634183 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:00.634264 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:00.634973 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:00.634977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:00.635171 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:00.635287 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.635293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.637089 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.637092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.638055 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.638070 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.639164 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.639171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.639326 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.640602 (+ 1276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:00.640621 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:00.642113 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:00.645035 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.645039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.645041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.645305 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.645654 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.645658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.645660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.645768 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.646268 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:00.646271 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:00.646366 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 12:38:00.646477 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 12:38:00.646607 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":3}
I20260619 12:38:00.647017 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.634264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.0.98.61:44579)
0619 12:38:00.634565 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:00.634578 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:00.634605 (+ 27us) server_negotiation.cc:407] Connection header received
0619 12:38:00.634641 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:00.634645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:00.634811 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:00.634984 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:00.635420 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.636654 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.638181 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.639028 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.639139 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.642868 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:00.642875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:00.642890 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:00.642914 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:00.644940 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.645438 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.645446 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.645447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.645540 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.645863 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.645865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.645866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.646164 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:00.646268 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:00.646621 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 12:38:00.646754 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":111,"threads_started":1}
I20260619 12:38:00.648418 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:38:00.649230 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:38:00.649947 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.649669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35447 (local address 127.0.0.1:58670)
0619 12:38:00.649815 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.649878 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 12:38:00.650740 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.650476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35447 (local address 127.0.0.1:58684)
0619 12:38:00.650646 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.650690 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260619 12:38:00.650882 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35447: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
I20260619 12:38:00.651361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35447
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33093,127.0.98.61:44579,127.0.98.60:35447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:38:00 dist-test-slave-kdrb krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 12:38:00.657248 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.649072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.0.98.61:44579)
0619 12:38:00.649213 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:00.649225 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:00.649243 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:38:00.649271 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:00.649273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:00.649305 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:00.649398 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:00.649937 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.650784 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.651649 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.651901 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.652010 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.654127 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:00.654130 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:00.654133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:00.654154 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:00.655676 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.656105 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.656110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.656112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.656183 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.656456 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.656459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.656461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.656628 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:00.656736 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:00.656965 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 12:38:00.657113 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":74}
I20260619 12:38:00.657248 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.648965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44579 (local address 127.0.0.1:34086)
0619 12:38:00.649090 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.649107 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:00.649193 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:00.649438 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:00.649441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:00.649662 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:00.649827 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.649833 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.650906 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.650909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.651549 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.651562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.652006 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.652009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.652100 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.652624 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:00.652636 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:00.653995 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:00.655813 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.655816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.655819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.656008 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.656273 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.656278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.656280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.656361 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.656716 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:00.656719 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:00.656797 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 12:38:00.656912 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 12:38:00.657055 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 12:38:00.658080 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:38:00.658869 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.658521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35447 (local address 127.0.0.1:58696)
0619 12:38:00.658673 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.658775 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 12:38:00.659545 13808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
I20260619 12:38:00.660523 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.660236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35447 (local address 127.0.0.1:58700)
0619 12:38:00.660376 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.660442 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 12:38:00.660750 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35447: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
I20260619 12:38:00.682075 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } attempt: 1
I20260619 12:38:00.683041 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.682700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35447 (local address 127.0.0.1:58714)
0619 12:38:00.682887 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.682954 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 12:38:00.683840 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.683639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35447 (local address 127.0.0.1:58726)
0619 12:38:00.683740 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.683787 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 12:38:00.683990 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35447: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
I20260619 12:38:00.710093 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } attempt: 1
I20260619 12:38:00.711097 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.710751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35447 (local address 127.0.0.1:58734)
0619 12:38:00.710939 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.711009 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 12:38:00.711957 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.711714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35447 (local address 127.0.0.1:58748)
0619 12:38:00.711829 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.711887 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 12:38:00.712105 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35447: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
I20260619 12:38:00.759141 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } attempt: 2
I20260619 12:38:00.760495 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.760001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35447 (local address 127.0.0.1:58762)
0619 12:38:00.760280 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.760370 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260619 12:38:00.761297 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.761083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35447 (local address 127.0.0.1:58774)
0619 12:38:00.761193 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.761239 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 12:38:00.761510 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35447: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
W20260619 12:38:00.769995 13865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:00.770278 13865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:00.770356 13865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:00.774564 13865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:38:00.774638 13865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:00.774658 13865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:00.774675 13865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:38:00.774732 13865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:38:00.780275 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } attempt: 2
I20260619 12:38:00.780381 13865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33093,127.0.98.61:44579,127.0.98.60:35447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:00.781157 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.780845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35447 (local address 127.0.0.1:58776)
0619 12:38:00.781007 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.781076 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 12:38:00.781648 13865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:00.781867 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.781650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35447 (local address 127.0.0.1:58784)
0619 12:38:00.781759 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.781813 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 12:38:00.781993 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35447: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35447: connect: Connection refused (error 111)
I20260619 12:38:00.782804 13865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:00.791949 13872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:00.791983 13865 server_base.cc:1034] running on GCE node
W20260619 12:38:00.791983 13871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:00.792032 13874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:00.792804 13865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:00.793988 13865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:00.795208 13865 hybrid_clock.cc:648] HybridClock initialized: now 1781872680795188 us; error 38 us; skew 500 ppm
Jun 19 12:38:00 dist-test-slave-kdrb krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:00.798205 13865 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 12:38:00.799224 13865 webserver.cc:460] Webserver started at http://127.0.98.60:33817/ using document root <none> and password file <none>
I20260619 12:38:00.799880 13865 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:00.799973 13865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:00.800216 13865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:38:00.802325 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/data/instance:
uuid: "d252a3bd9b834671b9c4f3135178346e"
format_stamp: "Formatted at 2026-06-19 12:38:00 on dist-test-slave-kdrb"
server_key: "ab1aa7b13f790d4a7b5ac9150eb9b9ac"
server_key_iv: "3c67176dee28ec62d1ed182c8a66c371"
server_key_version: "encryptionkey@0"
I20260619 12:38:00.802894 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/wal/instance:
uuid: "d252a3bd9b834671b9c4f3135178346e"
format_stamp: "Formatted at 2026-06-19 12:38:00 on dist-test-slave-kdrb"
server_key: "ab1aa7b13f790d4a7b5ac9150eb9b9ac"
server_key_iv: "3c67176dee28ec62d1ed182c8a66c371"
server_key_version: "encryptionkey@0"
I20260619 12:38:00.807027 13865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:38:00.809804 13881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:00.811195 13865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:38:00.811359 13865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/wal
uuid: "d252a3bd9b834671b9c4f3135178346e"
format_stamp: "Formatted at 2026-06-19 12:38:00 on dist-test-slave-kdrb"
server_key: "ab1aa7b13f790d4a7b5ac9150eb9b9ac"
server_key_iv: "3c67176dee28ec62d1ed182c8a66c371"
server_key_version: "encryptionkey@0"
I20260619 12:38:00.811530 13865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:00.844523 13865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:00.847702 13865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:00.847920 13865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:00.855146 13865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35447
I20260619 12:38:00.855165 13933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35447 every 8 connection(s)
I20260619 12:38:00.856246 13865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/data/info.pb
I20260619 12:38:00.858414 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 13865
I20260619 12:38:00.858563 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872516972220-392-0/minicluster-data/master-2/wal/instance
I20260619 12:38:00.858841 392 external_mini_cluster.cc:1428] Setting key 81308d9b155327605170e33f24939386
I20260619 12:38:00.859164 13934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:00.860664 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } attempt: 3
I20260619 12:38:00.864542 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:38:00 dist-test-slave-kdrb krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 12:38:00 dist-test-slave-kdrb krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 12:38:00 dist-test-slave-kdrb krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:38:00.873157 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } attempt: 3
I20260619 12:38:00.874001 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.861112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35447 (local address 127.0.0.1:58812)
0619 12:38:00.861270 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.861287 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:00.861386 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:00.862694 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:00.862697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:00.862917 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:00.863042 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.863049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.864883 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.864890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.865632 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.865675 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.866824 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.866829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.866910 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.867460 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:00.867519 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:00.869114 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:00.871613 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.871617 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.871619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.871875 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.872319 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.872322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.872323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.872399 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.873440 (+ 1041us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:00.873444 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:00.873557 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 12:38:00.873686 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 12:38:00.873820 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 12:38:00.875530 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.861361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.0.98.60:35447)
0619 12:38:00.861667 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:00.861673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:00.861685 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:38:00.861840 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:00.861864 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:00.862351 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:00.862740 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:00.863377 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.864729 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.865818 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.866698 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.866861 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.869287 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:00.869292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:00.869301 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:00.869325 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:00.871452 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.872010 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.872019 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.872028 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.872198 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.872497 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.872502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.872504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.873027 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:00.873177 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:00.874140 (+ 963us) server_negotiation.cc:299] Negotiation successful
0619 12:38:00.874466 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":358,"server-negotiator.queue_time_us":241,"spinlock_wait_cycles":22144,"thread_start_us":116,"threads_started":1}
I20260619 12:38:00.876648 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.860353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.0.98.60:35447)
0619 12:38:00.860857 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:00.860878 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:00.860924 (+ 46us) server_negotiation.cc:407] Connection header received
0619 12:38:00.861840 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:00.861864 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:00.862351 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:00.862740 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:00.864104 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.864826 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.865948 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.866694 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.866865 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.870340 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:00.870343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:00.870346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:00.870375 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:00.872140 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.872712 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.872715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.872717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.872787 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.873104 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.873109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.873112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.873337 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:00.873516 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:00.876335 (+ 2819us) server_negotiation.cc:299] Negotiation successful
0619 12:38:00.876466 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":339,"spinlock_wait_cycles":29184,"thread_start_us":145,"threads_started":1}
I20260619 12:38:00.878351 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.866551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33093 (local address 127.0.0.1:34084)
0619 12:38:00.867215 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.867267 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:00.867420 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:00.867802 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:00.867806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:00.868128 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:00.868241 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.868247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.868952 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.868958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.870034 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.870046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.870845 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.870847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.870934 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.871553 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:00.871564 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:00.874355 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:00.876279 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.876286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.876291 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.876602 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.877023 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.877026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.877028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.877112 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.877676 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:00.877682 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:00.877809 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 12:38:00.877948 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 12:38:00.878129 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"spinlock_wait_cycles":2816,"thread_start_us":106,"threads_started":1}
I20260619 12:38:00.878830 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.866598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.0.98.62:33093)
0619 12:38:00.866724 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:00.866729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:00.867334 (+ 605us) server_negotiation.cc:407] Connection header received
0619 12:38:00.867563 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:00.867567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:00.867618 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:00.867712 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:00.868356 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.868830 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.870211 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.870496 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.870628 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.874485 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:00.874489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:00.874493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:00.874517 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:00.876171 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.876833 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.876837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.876839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.876925 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.877222 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.877226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.877227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.877485 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:00.877606 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:00.878552 (+ 946us) server_negotiation.cc:299] Negotiation successful
0619 12:38:00.878678 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 12:38:00 dist-test-slave-kdrb krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 12:38:00.880283 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:38:00.881160 13784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668: Bootstrap starting.
I20260619 12:38:00.885082 13784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668: Neither blocks nor log segments found. Creating new log.
Jun 19 12:38:00 dist-test-slave-kdrb krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:38:00.886034 13784 log.cc:824] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:38:00 dist-test-slave-kdrb krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 12:38:00.886305 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.873602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35447 (local address 127.0.0.1:58814)
0619 12:38:00.873763 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.873780 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:00.873862 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:00.874836 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:00.874839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:00.875051 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:00.875177 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.875188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.876321 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.876326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.877084 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.877095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.877903 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.877907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.877991 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.878818 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:00.878834 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:00.880870 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:00.884567 (+ 3697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.884573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.884575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.884850 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.885178 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.885184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.885187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.885284 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.885755 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:00.885758 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:00.885865 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 12:38:00.885959 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 12:38:00.886096 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 12:38:00.886994 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.873889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.0.98.60:35447)
0619 12:38:00.874603 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:00.874615 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:00.874627 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:38:00.874660 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:00.874663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:00.874712 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:00.874810 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:00.875332 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.876185 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.877459 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.877770 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.878227 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.881445 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:00.881452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:00.881458 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:00.881481 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:00.884425 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.884977 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.884983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.884987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.885072 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.885402 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.885405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.885407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.885624 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:00.886590 (+ 966us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:00.886658 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 12:38:00.886813 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":89,"threads_started":1}
I20260619 12:38:00.889168 13784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668: No bootstrap required, opened a new log
I20260619 12:38:00.890406 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.881037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34098 (local address 127.0.98.61:44579)
0619 12:38:00.881154 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:00.881159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:00.881170 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:38:00.881201 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:00.881208 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:00.881241 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:00.881325 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:00.881767 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.882524 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.883670 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.884011 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.884112 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.886599 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:00.886603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:00.886608 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:00.886631 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:00.888463 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.888972 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.888979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.888983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.889069 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.889458 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.889463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.889467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.889705 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:00.889781 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:00.890139 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 12:38:00.890265 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:38:00.891037 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.880745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44579 (local address 127.0.0.1:34098)
0619 12:38:00.880993 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.881008 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:00.881070 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:00.881340 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:00.881343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:00.881518 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:00.881628 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.881636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.882657 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.882660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.883526 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.883538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.884314 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.884317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.884414 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.884869 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:00.884889 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:00.886453 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:00.888585 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.888589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.888591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.888846 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.889201 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.889204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.889206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.889354 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.889859 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:00.889862 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:00.889954 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 12:38:00.890739 (+ 785us) client_negotiation.cc:240] Negotiation successful
0619 12:38:00.890895 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260619 12:38:00.891193 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0: Bootstrap starting.
I20260619 12:38:00.892470 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:38:00.892867 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.878803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34092 (local address 127.0.98.62:33093)
0619 12:38:00.879240 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:00.879256 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:00.880006 (+ 750us) server_negotiation.cc:407] Connection header received
0619 12:38:00.880049 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:00.880052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:00.880104 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:00.880188 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:00.880980 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.881829 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.882898 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.883240 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.883348 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.887869 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:00.887872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:00.887875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:00.887900 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:00.889594 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.890069 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.890073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.890075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.890161 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.890473 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.890476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.890478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.890677 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:00.892001 (+ 1324us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:00.892572 (+ 571us) server_negotiation.cc:299] Negotiation successful
0619 12:38:00.892697 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":71,"threads_started":1}
I20260619 12:38:00.893772 13784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } }
I20260619 12:38:00.893992 13784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:00.894055 13784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5099accf4e02456a92a9c1961b07c668, State: Initialized, Role: FOLLOWER
I20260619 12:38:00.894227 13859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:00.894585 13784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } }
I20260619 12:38:00.895123 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } } }
I20260619 12:38:00.895255 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:38:00.896032 13784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:38:00.896163 13859 log.cc:824] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:00.898583 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0: No bootstrap required, opened a new log
Jun 19 12:38:00 dist-test-slave-kdrb krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 12:38:00 dist-test-slave-kdrb krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 12:38:00.903195 13859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } }
I20260619 12:38:00.903628 13859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:00.903735 13859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d987fd1d88543ee99790d1da1cd10b0, State: Initialized, Role: FOLLOWER
I20260619 12:38:00.904321 13859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } }
I20260619 12:38:00.901113 13716 master_runner.cc:186] Error getting master registration for 127.0.98.61:44579: OK, instance_id { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" instance_seqno: 1781872680618670 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:38:00.905284 13716 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:38:00.905539 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } } }
I20260619 12:38:00.905678 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:38:00.906313 13859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:38:00.907410 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.894775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35447 (local address 127.0.0.1:58816)
0619 12:38:00.896676 (+ 1901us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:00.896702 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:00.896817 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:00.897102 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:00.897104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:00.897280 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:00.897365 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.897372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.898306 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.898312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.899665 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:00.899680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.900425 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.900428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:00.900520 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.901041 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:00.901063 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:00.903194 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:00.905271 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.905274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.905276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.905493 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.905767 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:00.905769 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:00.905771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:00.905848 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:00.906530 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:00.906533 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:00.906635 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 12:38:00.906782 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 12:38:00.907146 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1824}
I20260619 12:38:00.907433 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.894976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.0.98.60:35447)
0619 12:38:00.895073 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:00.895077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:00.896755 (+ 1678us) server_negotiation.cc:407] Connection header received
0619 12:38:00.896909 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:00.896912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:00.896950 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:00.896993 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:00.897473 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.898181 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.899821 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.900198 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.900282 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.903337 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:00.903341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:00.903344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:00.903369 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:00.905148 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.905587 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.905590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.905592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.905661 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.905965 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.905969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.905971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.906212 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:00.906498 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:00.906874 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 12:38:00.907254 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":225,"server-negotiator.queue_time_us":38}
W20260619 12:38:00.908826 13955 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:38:00.908931 13955 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:38:00.910442 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.894537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.0.98.61:44579)
0619 12:38:00.895019 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:00.895025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:00.895039 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:38:00.895076 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:00.895079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:00.895126 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:00.895235 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:00.896006 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.896748 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.897830 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.898235 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.898698 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.901418 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:00.901423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:00.901426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:00.901451 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:00.903196 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.903820 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.903824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.903827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.903927 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.904289 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.904295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.904300 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.904585 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:00.909996 (+ 5411us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:00.910117 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 12:38:00.910272 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400}
I20260619 12:38:00.911178 13934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e: Bootstrap starting.
W20260619 12:38:00.913486 13968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:38:00.913636 13968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:38:00.914425 13934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:00.915206 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:38:00.915586 13934 log.cc:824] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:00.917444 13791 master_runner.cc:186] Error getting master registration for 127.0.98.60:35447: OK, instance_id { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" instance_seqno: 1781872680853033 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:38:00.920996 13791 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:38:00.928613 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.917529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34094 (local address 127.0.98.62:33093)
0619 12:38:00.917671 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:00.917678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:00.918877 (+ 1199us) server_negotiation.cc:407] Connection header received
0619 12:38:00.919066 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:00.919069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:00.919124 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:00.919177 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:00.920254 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.921083 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.922407 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.922790 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.922908 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.924791 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:00.924795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:00.924799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:00.924829 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:00.926616 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.927124 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.927127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.927130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.927232 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.927562 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.927565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.927567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.927802 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:00.927942 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:00.928304 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 12:38:00.928441 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 12:38:00.932890 13934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e: No bootstrap required, opened a new log
I20260619 12:38:00.937532 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.921695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.0.98.60:35447)
0619 12:38:00.921834 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:00.921841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:00.928111 (+ 6270us) server_negotiation.cc:407] Connection header received
0619 12:38:00.928294 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:00.928297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:00.928360 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:00.928416 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:00.929316 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.930126 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.931513 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.931858 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.931937 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.933837 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:00.933842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:00.933845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:00.933872 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:00.935542 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.936104 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.936108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.936110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.936213 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.936516 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.936519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.936521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.936742 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:00.936860 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:00.937226 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 12:38:00.937363 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:38:00.938683 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:00.918260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34108 (local address 127.0.98.61:44579)
0619 12:38:00.918363 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:00.918367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:00.927762 (+ 9395us) server_negotiation.cc:407] Connection header received
0619 12:38:00.927912 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:00.927916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:00.927971 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:00.928048 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:00.929386 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.930192 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.931394 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:00.931750 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:00.931831 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:00.933984 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:00.933988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:00.933991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:00.934021 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:00.936338 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.936858 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.936863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.936865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.936954 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:00.937269 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:00.937273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:00.937275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:00.937543 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:00.937698 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:00.938409 (+ 711us) server_negotiation.cc:299] Negotiation successful
0619 12:38:00.938529 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 12:38:00.938809 13934 raft_consensus.cc:348] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } }
I20260619 12:38:00.939117 13934 raft_consensus.cc:374] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:00.939188 13934 raft_consensus.cc:729] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d252a3bd9b834671b9c4f3135178346e, State: Initialized, Role: FOLLOWER
I20260619 12:38:00.939751 13934 consensus_queue.cc:260] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } }
I20260619 12:38:00.940299 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } } }
I20260619 12:38:00.940492 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:38:00.944779 13934 sys_catalog.cc:564] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e [sys.catalog]: configured and running, proceeding with master startup.
W20260619 12:38:00.950287 13989 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:38:00.950378 13989 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:38:00.954272 13865 master_runner.cc:263] No leader master found from master d252a3bd9b834671b9c4f3135178346e
I20260619 12:38:00.954352 13865 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 12:38:01.099395 13887 tablet.cc:2369] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:38:01.115065 13812 tablet.cc:2369] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:38:01.124341 13956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:38:01.124563 13956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } }
I20260619 12:38:01.125631 13956 leader_election.cc:288] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5099accf4e02456a92a9c1961b07c668 (127.0.98.62:33093), d252a3bd9b834671b9c4f3135178346e (127.0.98.60:35447)
I20260619 12:38:01.126222 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d987fd1d88543ee99790d1da1cd10b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5099accf4e02456a92a9c1961b07c668" is_pre_election: true
I20260619 12:38:01.126276 13908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d987fd1d88543ee99790d1da1cd10b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d252a3bd9b834671b9c4f3135178346e" is_pre_election: true
I20260619 12:38:01.126550 13758 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d987fd1d88543ee99790d1da1cd10b0 in term 0.
I20260619 12:38:01.126596 13908 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d987fd1d88543ee99790d1da1cd10b0 in term 0.
I20260619 12:38:01.127110 13808 leader_election.cc:302] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d987fd1d88543ee99790d1da1cd10b0, d252a3bd9b834671b9c4f3135178346e; no voters:
I20260619 12:38:01.127424 13956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:38:01.127523 13956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:38:01.127573 13956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:01.128832 13956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } }
I20260619 12:38:01.129225 13956 leader_election.cc:288] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [CANDIDATE]: Term 1 election: Requested vote from peers 5099accf4e02456a92a9c1961b07c668 (127.0.98.62:33093), d252a3bd9b834671b9c4f3135178346e (127.0.98.60:35447)
I20260619 12:38:01.129606 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d987fd1d88543ee99790d1da1cd10b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5099accf4e02456a92a9c1961b07c668"
I20260619 12:38:01.129761 13758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:01.131222 13758 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d987fd1d88543ee99790d1da1cd10b0 in term 1.
I20260619 12:38:01.131583 13809 leader_election.cc:302] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d987fd1d88543ee99790d1da1cd10b0, 5099accf4e02456a92a9c1961b07c668; no voters:
I20260619 12:38:01.131623 13908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d987fd1d88543ee99790d1da1cd10b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d252a3bd9b834671b9c4f3135178346e"
I20260619 12:38:01.131753 13908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:01.131804 13956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:38:01.132019 13956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d987fd1d88543ee99790d1da1cd10b0, State: Running, Role: LEADER
I20260619 12:38:01.132364 13956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } }
I20260619 12:38:01.132982 13908 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d987fd1d88543ee99790d1da1cd10b0 in term 1.
I20260619 12:38:01.134076 13991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d987fd1d88543ee99790d1da1cd10b0. Latest consensus state: current_term: 1 leader_uuid: "4d987fd1d88543ee99790d1da1cd10b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } } }
I20260619 12:38:01.134223 13991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [sys.catalog]: This master's current role is: LEADER
W20260619 12:38:01.134486 13737 tablet.cc:2369] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:38:01.135411 13993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:38:01.136300 13993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:38:01.142024 13908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e [term 1 FOLLOWER]: Refusing update from remote peer 4d987fd1d88543ee99790d1da1cd10b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:38:01.142058 13758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668 [term 1 FOLLOWER]: Refusing update from remote peer 4d987fd1d88543ee99790d1da1cd10b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:38:01.142693 13956 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:38:01.142889 13991 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:38:01.147127 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d987fd1d88543ee99790d1da1cd10b0. Latest consensus state: current_term: 1 leader_uuid: "4d987fd1d88543ee99790d1da1cd10b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } } }
I20260619 12:38:01.147168 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d987fd1d88543ee99790d1da1cd10b0. Latest consensus state: current_term: 1 leader_uuid: "4d987fd1d88543ee99790d1da1cd10b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } } }
I20260619 12:38:01.147286 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:38:01.147305 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:38:01.153580 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d987fd1d88543ee99790d1da1cd10b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } } }
I20260619 12:38:01.153628 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d987fd1d88543ee99790d1da1cd10b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } } }
I20260619 12:38:01.153762 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:38:01.153760 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:38:01.154402 13999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d987fd1d88543ee99790d1da1cd10b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } } }
I20260619 12:38:01.154451 13991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d987fd1d88543ee99790d1da1cd10b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099accf4e02456a92a9c1961b07c668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33093 } } peers { permanent_uuid: "4d987fd1d88543ee99790d1da1cd10b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44579 } } peers { permanent_uuid: "d252a3bd9b834671b9c4f3135178346e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35447 } } }
I20260619 12:38:01.154628 13999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [sys.catalog]: This master's current role is: LEADER
I20260619 12:38:01.154670 13991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0 [sys.catalog]: This master's current role is: LEADER
I20260619 12:38:01.155594 13993 catalog_manager.cc:1347] Generated new cluster ID: 740ea09838fb49dc8fab435402ee98b7
I20260619 12:38:01.155664 13993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:38:01.173023 13993 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:38:01.174270 13993 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:38:01.185499 13993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d987fd1d88543ee99790d1da1cd10b0: Generated new TSK 0
I20260619 12:38:01.186465 13993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:38:01.910321 13955 catalog_manager.cc:1259] Loaded cluster ID: 740ea09838fb49dc8fab435402ee98b7
I20260619 12:38:01.910414 13955 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668: loading cluster ID for follower catalog manager: success
I20260619 12:38:01.912238 13955 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668: acquiring CA information for follower catalog manager: success
I20260619 12:38:01.913529 13955 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5099accf4e02456a92a9c1961b07c668: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:38:01.951750 13989 catalog_manager.cc:1259] Loaded cluster ID: 740ea09838fb49dc8fab435402ee98b7
I20260619 12:38:01.951835 13989 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e: loading cluster ID for follower catalog manager: success
I20260619 12:38:01.953536 13989 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e: acquiring CA information for follower catalog manager: success
I20260619 12:38:01.954771 13989 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d252a3bd9b834671b9c4f3135178346e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:38:02.259229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 13716
I20260619 12:38:02.265502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 13791
I20260619 12:38:02.271543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 13865
2026-06-19T12:38:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3231 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:38:02 dist-test-slave-kdrb krb5kdc[14009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:38:02 dist-test-slave-kdrb krb5kdc[14009](info): set up 2 sockets
Jun 19 12:38:02 dist-test-slave-kdrb krb5kdc[14009](info): commencing operation
krb5kdc: starting...
W20260619 12:38:03.315881 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:38:03 dist-test-slave-kdrb krb5kdc[14009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:38:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:38:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:38:03.442865 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41465
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:38:03.555240 14022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:03.555606 14022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:03.555698 14022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:03.559821 14022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:38:03.559930 14022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:03.559970 14022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:03.560009 14022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:38:03.560042 14022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:38:03.565522 14022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:03.566689 14022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:03.567766 14022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:03.576368 14027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:03.576380 14030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:03.576431 14022 server_base.cc:1034] running on GCE node
W20260619 12:38:03.576391 14028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:03.577298 14022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:03.578449 14022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:03.579690 14022 hybrid_clock.cc:648] HybridClock initialized: now 1781872683579674 us; error 39 us; skew 500 ppm
Jun 19 12:38:03 dist-test-slave-kdrb krb5kdc[14009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:03.582588 14022 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:38:03.583559 14022 webserver.cc:460] Webserver started at http://127.0.98.62:38819/ using document root <none> and password file <none>
I20260619 12:38:03.584139 14022 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:03.584246 14022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:03.584476 14022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:38:03.586535 14022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "94362d2316f04bff9f0ebb07e2bb153c"
format_stamp: "Formatted at 2026-06-19 12:38:03 on dist-test-slave-kdrb"
server_key: "01e436e7c0b3fcb74e727f4fd7948ecb"
server_key_iv: "2a99ac6c7a5344384b98df50d3b151c1"
server_key_version: "encryptionkey@0"
I20260619 12:38:03.587100 14022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "94362d2316f04bff9f0ebb07e2bb153c"
format_stamp: "Formatted at 2026-06-19 12:38:03 on dist-test-slave-kdrb"
server_key: "01e436e7c0b3fcb74e727f4fd7948ecb"
server_key_iv: "2a99ac6c7a5344384b98df50d3b151c1"
server_key_version: "encryptionkey@0"
I20260619 12:38:03.590952 14022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:38:03.593583 14037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:03.594995 14022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:38:03.595140 14022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "94362d2316f04bff9f0ebb07e2bb153c"
format_stamp: "Formatted at 2026-06-19 12:38:03 on dist-test-slave-kdrb"
server_key: "01e436e7c0b3fcb74e727f4fd7948ecb"
server_key_iv: "2a99ac6c7a5344384b98df50d3b151c1"
server_key_version: "encryptionkey@0"
I20260619 12:38:03.595268 14022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:03.621299 14022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:03.624891 14022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:03.625115 14022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:03.632652 14022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41465
I20260619 12:38:03.632673 14089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41465 every 8 connection(s)
I20260619 12:38:03.633680 14022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:38:03.636521 14090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:03.639811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 14022
I20260619 12:38:03.639948 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:38:03.640240 392 external_mini_cluster.cc:1428] Setting key 2bce1ccdea99d69d64585565fdbea4e1
I20260619 12:38:03.642767 14090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c: Bootstrap starting.
I20260619 12:38:03.645092 14090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:03.645852 14090 log.cc:824] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:03.647851 14090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c: No bootstrap required, opened a new log
Jun 19 12:38:03 dist-test-slave-kdrb krb5kdc[14009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:38:03.652120 14090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94362d2316f04bff9f0ebb07e2bb153c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41465 } }
I20260619 12:38:03.652462 14090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:03.652529 14090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94362d2316f04bff9f0ebb07e2bb153c, State: Initialized, Role: FOLLOWER
I20260619 12:38:03.653035 14090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94362d2316f04bff9f0ebb07e2bb153c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41465 } }
I20260619 12:38:03.653241 14090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:38:03.653316 14090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:38:03.653422 14090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:03.654734 14090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94362d2316f04bff9f0ebb07e2bb153c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41465 } }
I20260619 12:38:03.655138 14090 leader_election.cc:302] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94362d2316f04bff9f0ebb07e2bb153c; no voters:
I20260619 12:38:03.655511 14090 leader_election.cc:288] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:38:03.655666 14095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:38:03.655915 14095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [term 1 LEADER]: Becoming Leader. State: Replica: 94362d2316f04bff9f0ebb07e2bb153c, State: Running, Role: LEADER
I20260619 12:38:03.656121 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:03.641672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.0.98.62:41465)
0619 12:38:03.642147 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:03.642167 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:03.642202 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:38:03.642868 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:03.642891 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:03.643247 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:03.643617 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:03.644435 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:03.645504 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:03.646440 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:03.646949 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:03.647125 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:03.649914 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:03.649920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:03.649940 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:03.649973 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:03.652640 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:03.653160 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:03.653169 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:03.653176 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:03.653337 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:03.653678 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:03.653684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:03.653688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:03.654227 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:03.654435 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:03.654653 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 12:38:03.655015 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":145,"threads_started":1}
I20260619 12:38:03.656265 14095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94362d2316f04bff9f0ebb07e2bb153c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41465 } }
I20260619 12:38:03.656836 14090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:38:03.658021 14097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94362d2316f04bff9f0ebb07e2bb153c. Latest consensus state: current_term: 1 leader_uuid: "94362d2316f04bff9f0ebb07e2bb153c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94362d2316f04bff9f0ebb07e2bb153c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41465 } } }
I20260619 12:38:03.658146 14097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [sys.catalog]: This master's current role is: LEADER
I20260619 12:38:03.658406 14096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94362d2316f04bff9f0ebb07e2bb153c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94362d2316f04bff9f0ebb07e2bb153c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41465 } } }
I20260619 12:38:03.658509 14096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c [sys.catalog]: This master's current role is: LEADER
I20260619 12:38:03.659641 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 12:38:03.662400 14110 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:38:03.662475 14110 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:38:03.662560 14104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:38:03.663286 14104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:38:03.670060 14104 catalog_manager.cc:1347] Generated new cluster ID: 5ae8cd3d3eb348e9b0d527cbd2577fa1
I20260619 12:38:03.670174 14104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:38:03.685397 14104 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:38:03.686419 14104 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:38:03.693826 14104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94362d2316f04bff9f0ebb07e2bb153c: Generated new TSK 0
I20260619 12:38:03.694574 14104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:38:03.716014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 14022
2026-06-19T12:38:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1443 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:38:03 dist-test-slave-kdrb krb5kdc[14117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:38:03 dist-test-slave-kdrb krb5kdc[14117](info): set up 2 sockets
Jun 19 12:38:03 dist-test-slave-kdrb krb5kdc[14117](info): commencing operation
krb5kdc: starting...
W20260619 12:38:04.763741 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:38:04 dist-test-slave-kdrb krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:38:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:38:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:38:04.893992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42511
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:38:05.005155 14130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:05.005484 14130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:05.005527 14130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:05.010074 14130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:38:05.010165 14130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:05.010188 14130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:05.010206 14130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:38:05.010222 14130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:38:05.016214 14130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:05.017360 14130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:05.018419 14130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:05.027769 14135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:05.027769 14138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:05.027815 14136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:05.028045 14130 server_base.cc:1034] running on GCE node
I20260619 12:38:05.028705 14130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:05.029924 14130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:05.031090 14130 hybrid_clock.cc:648] HybridClock initialized: now 1781872685031057 us; error 51 us; skew 500 ppm
Jun 19 12:38:05 dist-test-slave-kdrb krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:05.039686 14130 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:38:05.040778 14130 webserver.cc:460] Webserver started at http://127.0.98.62:39205/ using document root <none> and password file <none>
I20260619 12:38:05.041424 14130 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:05.041498 14130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:05.041731 14130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:38:05.044186 14130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "69c5c649c01847158f9a7230870bd4bc"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "6f26face1203a3ab3a909f799f4a5ddb"
server_key_iv: "1cfa330974990f42586ef0e6987837ef"
server_key_version: "encryptionkey@0"
I20260619 12:38:05.044839 14130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "69c5c649c01847158f9a7230870bd4bc"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "6f26face1203a3ab3a909f799f4a5ddb"
server_key_iv: "1cfa330974990f42586ef0e6987837ef"
server_key_version: "encryptionkey@0"
I20260619 12:38:05.048826 14130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:38:05.051527 14145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:05.052827 14130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:38:05.052966 14130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "69c5c649c01847158f9a7230870bd4bc"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "6f26face1203a3ab3a909f799f4a5ddb"
server_key_iv: "1cfa330974990f42586ef0e6987837ef"
server_key_version: "encryptionkey@0"
I20260619 12:38:05.053082 14130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:05.067735 14130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:05.070736 14130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:05.070953 14130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:05.078303 14130 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42511
I20260619 12:38:05.078603 14197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42511 every 8 connection(s)
I20260619 12:38:05.079388 14130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:38:05.080529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 14130
I20260619 12:38:05.080658 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:38:05.080978 392 external_mini_cluster.cc:1428] Setting key 450cd0e43829898110bab553b56077f1
I20260619 12:38:05.082809 14198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:05.089737 14198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc: Bootstrap starting.
Jun 19 12:38:05 dist-test-slave-kdrb krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:38:05.092278 14198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:05.093070 14198 log.cc:824] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:05.095566 14198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc: No bootstrap required, opened a new log
I20260619 12:38:05.096208 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:05.082388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.0.98.62:42511)
0619 12:38:05.082832 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:05.082870 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:05.082918 (+ 48us) server_negotiation.cc:407] Connection header received
0619 12:38:05.083647 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:05.083666 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:05.084017 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:05.084313 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:05.085239 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.086258 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.087429 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.087997 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.088120 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:05.090975 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:05.090980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:05.090994 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:05.091024 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:05.092972 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:05.093496 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:05.093528 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:05.093539 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:05.093710 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:05.094011 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:05.094015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:05.094018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:05.094438 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:05.094586 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:05.094975 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 12:38:05.095329 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":137,"threads_started":1}
I20260619 12:38:05.099092 14198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } }
I20260619 12:38:05.099328 14198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:05.099365 14198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c5c649c01847158f9a7230870bd4bc, State: Initialized, Role: FOLLOWER
I20260619 12:38:05.099850 14198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } }
I20260619 12:38:05.099973 14198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:38:05.100077 14198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:38:05.100195 14198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:05.101186 14198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } }
I20260619 12:38:05.101547 14198 leader_election.cc:302] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69c5c649c01847158f9a7230870bd4bc; no voters:
I20260619 12:38:05.101899 14198 leader_election.cc:288] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:38:05.102033 14203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:38:05.102344 14203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 1 LEADER]: Becoming Leader. State: Replica: 69c5c649c01847158f9a7230870bd4bc, State: Running, Role: LEADER
I20260619 12:38:05.102700 14203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } }
I20260619 12:38:05.103284 14198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:38:05.104662 14205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69c5c649c01847158f9a7230870bd4bc. Latest consensus state: current_term: 1 leader_uuid: "69c5c649c01847158f9a7230870bd4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } }
I20260619 12:38:05.104666 14204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69c5c649c01847158f9a7230870bd4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } }
I20260619 12:38:05.104877 14204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [sys.catalog]: This master's current role is: LEADER
I20260619 12:38:05.105122 14205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [sys.catalog]: This master's current role is: LEADER
I20260619 12:38:05.105382 14214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:38:05.108382 14214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:38:05.113739 14214 catalog_manager.cc:1347] Generated new cluster ID: 1978d65ee6fe40d0bf0abeaea1ef3c82
I20260619 12:38:05.113827 14214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:38:05.125569 14214 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:38:05.126665 14214 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:38:05.138572 14214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc: Generated new TSK 0
I20260619 12:38:05.139338 14214 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:38:05.173969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42511
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:38:05.288117 14224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:05.288394 14224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:05.288439 14224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:05.292606 14224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:05.292676 14224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:05.292773 14224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:38:05.298071 14224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:05.299227 14224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:05.300374 14224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:05.310345 14230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:05.310415 14229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:05.310487 14224 server_base.cc:1034] running on GCE node
W20260619 12:38:05.310359 14232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:05.311098 14224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:05.311913 14224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:05.313079 14224 hybrid_clock.cc:648] HybridClock initialized: now 1781872685313062 us; error 37 us; skew 500 ppm
Jun 19 12:38:05 dist-test-slave-kdrb krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:05.316093 14224 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:38:05.317020 14224 webserver.cc:460] Webserver started at http://127.0.98.1:42059/ using document root <none> and password file <none>
I20260619 12:38:05.317603 14224 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:05.317695 14224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:05.317932 14224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:38:05.320168 14224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "5528c98868d005bec62fb8bd75f80dac"
server_key_iv: "d19762f5e07ab6c00848edf20c0a4a8d"
server_key_version: "encryptionkey@0"
I20260619 12:38:05.320763 14224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "5528c98868d005bec62fb8bd75f80dac"
server_key_iv: "d19762f5e07ab6c00848edf20c0a4a8d"
server_key_version: "encryptionkey@0"
I20260619 12:38:05.324620 14224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 12:38:05.327175 14239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:05.328524 14224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 12:38:05.328668 14224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "5528c98868d005bec62fb8bd75f80dac"
server_key_iv: "d19762f5e07ab6c00848edf20c0a4a8d"
server_key_version: "encryptionkey@0"
I20260619 12:38:05.328789 14224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:05.383414 14224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:05.386688 14224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:05.387010 14224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:05.387669 14224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:38:05.388525 14224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:38:05.388598 14224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:05.388671 14224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:38:05.388731 14224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:05.397876 14224 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39831
I20260619 12:38:05.397899 14352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39831 every 8 connection(s)
I20260619 12:38:05.398826 14224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:38:05.401310 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 14224
I20260619 12:38:05.401463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:38:05.401775 392 external_mini_cluster.cc:1428] Setting key 7f02e3a242fa2f94ec0592975fd22786
Jun 19 12:38:05 dist-test-slave-kdrb krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:38:05.415068 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:05.400773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50761 (local address 127.0.98.62:42511)
0619 12:38:05.400913 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:05.400916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:05.401973 (+ 1057us) server_negotiation.cc:407] Connection header received
0619 12:38:05.403816 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:05.403820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:05.403883 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:05.404004 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:05.405556 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.406088 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.407317 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.407799 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.407924 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:05.410761 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:05.410764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:05.410766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:05.410791 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:05.412455 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:05.413092 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:05.413095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:05.413096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:05.413186 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:05.413591 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:05.413615 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:05.413616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:05.413835 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:05.413923 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:05.414649 (+ 726us) server_negotiation.cc:299] Negotiation successful
0619 12:38:05.414805 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:38:05.415856 14355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:05.401183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:50761)
0619 12:38:05.401773 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:05.401839 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:05.403560 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:05.404222 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:05.404230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:05.404617 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:05.405394 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:05.405417 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.406214 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.406217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:05.407176 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:05.407185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.407987 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.407990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:05.408169 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:05.408877 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:05.408894 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:05.410558 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:05.412677 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:05.412682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:05.412697 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:05.412977 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:05.413288 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:05.413290 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:05.413291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:05.413484 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:05.413954 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:05.413959 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:05.414337 (+ 378us) client_negotiation.cc:769] Sending connection context
0619 12:38:05.414548 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 12:38:05.414838 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":171,"threads_started":1}
I20260619 12:38:05.417304 14353 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42511
I20260619 12:38:05.417569 14353 heartbeater.cc:464] Registering TS with master...
I20260619 12:38:05.418088 14353 heartbeater.cc:511] Master 127.0.98.62:42511 requested a full tablet report, sending...
I20260619 12:38:05.419775 14162 ts_manager.cc:194] Registered new tserver with Master: fe8ea6b60ebb44268af0f3c9ff0d90ac (127.0.98.1:39831)
I20260619 12:38:05.420907 14162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:38:05.430751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42511
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:38:05.545189 14358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:05.545516 14358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:05.545591 14358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:05.549714 14358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:05.549829 14358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:05.549932 14358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:38:05.555644 14358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:05.556790 14358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:05.557924 14358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:05.567836 14366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:05.567836 14364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:05.567836 14363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:05.568127 14358 server_base.cc:1034] running on GCE node
I20260619 12:38:05.568656 14358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:05.569408 14358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:05.570631 14358 hybrid_clock.cc:648] HybridClock initialized: now 1781872685570617 us; error 36 us; skew 500 ppm
Jun 19 12:38:05 dist-test-slave-kdrb krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:05.573941 14358 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:38:05.574941 14358 webserver.cc:460] Webserver started at http://127.0.98.2:43471/ using document root <none> and password file <none>
I20260619 12:38:05.575603 14358 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:05.575698 14358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:05.575937 14358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:38:05.578066 14358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "da0a02cfb2b04eb685bd7503ff955c25"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "b7ad49f910b323a226f629ea7481103c"
server_key_iv: "e7de355d38f42f4737a69c9821de6e03"
server_key_version: "encryptionkey@0"
I20260619 12:38:05.578653 14358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "da0a02cfb2b04eb685bd7503ff955c25"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "b7ad49f910b323a226f629ea7481103c"
server_key_iv: "e7de355d38f42f4737a69c9821de6e03"
server_key_version: "encryptionkey@0"
I20260619 12:38:05.582670 14358 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:38:05.585464 14373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:05.586838 14358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 12:38:05.587003 14358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "da0a02cfb2b04eb685bd7503ff955c25"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "b7ad49f910b323a226f629ea7481103c"
server_key_iv: "e7de355d38f42f4737a69c9821de6e03"
server_key_version: "encryptionkey@0"
I20260619 12:38:05.587143 14358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:05.614553 14358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:05.617805 14358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:05.618031 14358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:05.618666 14358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:38:05.619572 14358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:38:05.619638 14358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:05.619709 14358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:38:05.619755 14358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:05.629031 14358 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37989
I20260619 12:38:05.629066 14486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37989 every 8 connection(s)
I20260619 12:38:05.629976 14358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:38:05.637970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 14358
I20260619 12:38:05.638111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:38:05.638465 392 external_mini_cluster.cc:1428] Setting key 9d8763d33a9909880cdc03c05eab3a16
Jun 19 12:38:05 dist-test-slave-kdrb krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:38:05.645771 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:05.631904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58255 (local address 127.0.98.62:42511)
0619 12:38:05.632110 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:05.632114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:05.633137 (+ 1023us) server_negotiation.cc:407] Connection header received
0619 12:38:05.634579 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:05.634582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:05.634651 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:05.634754 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:05.636102 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.636800 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.638396 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.638697 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.638832 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:05.641710 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:05.641714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:05.641716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:05.641744 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:05.643273 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:05.643854 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:05.643859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:05.643861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:05.643964 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:05.644432 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:05.644435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:05.644436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:05.644613 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:05.644761 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:05.645361 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 12:38:05.645513 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260619 12:38:05.646477 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:05.632350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:58255)
0619 12:38:05.632927 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:05.633026 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:05.634349 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:05.634960 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:05.634968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:05.635364 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:05.635955 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:05.635970 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.636937 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.636940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:05.638266 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:05.638275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.638874 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.638878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:05.639091 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:05.639766 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:05.639805 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:05.641539 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:05.643398 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:05.643404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:05.643421 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:05.643751 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:05.644093 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:05.644099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:05.644103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:05.644325 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:05.644790 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:05.644796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:05.645078 (+ 282us) client_negotiation.cc:769] Sending connection context
0619 12:38:05.645249 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 12:38:05.645524 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260619 12:38:05.647805 14487 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42511
I20260619 12:38:05.648064 14487 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:38:05.648610 14487 heartbeater.cc:511] Master 127.0.98.62:42511 requested a full tablet report, sending...
I20260619 12:38:05.649797 14162 ts_manager.cc:194] Registered new tserver with Master: da0a02cfb2b04eb685bd7503ff955c25 (127.0.98.2:37989)
I20260619 12:38:05.650491 14162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:38:05.666790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42511
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:38:05.779870 14492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:05.780184 14492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:05.780272 14492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:05.784919 14492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:05.784996 14492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:05.785100 14492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:38:05.790732 14492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:05.791981 14492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:05.793073 14492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:05.803117 14498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:05.803141 14497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:05.803248 14492 server_base.cc:1034] running on GCE node
W20260619 12:38:05.803452 14500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:05.803990 14492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:05.804683 14492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:05.805851 14492 hybrid_clock.cc:648] HybridClock initialized: now 1781872685805834 us; error 32 us; skew 500 ppm
Jun 19 12:38:05 dist-test-slave-kdrb krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:05.808768 14492 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:38:05.809643 14492 webserver.cc:460] Webserver started at http://127.0.98.3:42989/ using document root <none> and password file <none>
I20260619 12:38:05.810174 14492 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:05.810235 14492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:05.810401 14492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:38:05.812529 14492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "869686f8889749978e8fb1bca3e3b729"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "701a7555e02c7e40eeb973f5e023011a"
server_key_iv: "3f5df402b22e936c5e082756ace7db9c"
server_key_version: "encryptionkey@0"
I20260619 12:38:05.813069 14492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "869686f8889749978e8fb1bca3e3b729"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "701a7555e02c7e40eeb973f5e023011a"
server_key_iv: "3f5df402b22e936c5e082756ace7db9c"
server_key_version: "encryptionkey@0"
I20260619 12:38:05.817006 14492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:38:05.819734 14507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:05.821075 14492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:38:05.821234 14492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "869686f8889749978e8fb1bca3e3b729"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "701a7555e02c7e40eeb973f5e023011a"
server_key_iv: "3f5df402b22e936c5e082756ace7db9c"
server_key_version: "encryptionkey@0"
I20260619 12:38:05.821365 14492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:05.838883 14492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:05.842475 14492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:05.842743 14492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:05.843400 14492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:38:05.844398 14492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:38:05.844478 14492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:05.844583 14492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:38:05.844640 14492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:05.854168 14492 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42635
I20260619 12:38:05.854184 14620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42635 every 8 connection(s)
I20260619 12:38:05.855154 14492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:38:05.863644 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 14492
I20260619 12:38:05.863756 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:38:05.864019 392 external_mini_cluster.cc:1428] Setting key 5a305f7fca06546ac49359dfca092b30
Jun 19 12:38:05 dist-test-slave-kdrb krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:38:05.869941 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:05.857092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59713 (local address 127.0.98.62:42511)
0619 12:38:05.857240 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:05.857245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:05.858265 (+ 1020us) server_negotiation.cc:407] Connection header received
0619 12:38:05.859627 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:05.859630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:05.859679 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:05.859768 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:05.861085 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.861637 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.862964 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.863202 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.863373 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:05.866352 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:05.866356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:05.866359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:05.866393 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:05.867819 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:05.868365 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:05.868374 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:05.868375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:05.868450 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:05.868822 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:05.868825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:05.868827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:05.869013 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:05.869126 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:05.869649 (+ 523us) server_negotiation.cc:299] Negotiation successful
0619 12:38:05.869778 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:38:05.870832 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:05.857592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:59713)
0619 12:38:05.858106 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:05.858151 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:05.859340 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:05.859939 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:05.859946 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:05.860325 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:05.860900 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:05.860917 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.861784 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.861787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:05.862811 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:05.862821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.863357 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.863359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:05.863576 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:05.864303 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:05.864321 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:05.866198 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:05.867944 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:05.867950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:05.867967 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:05.868245 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:05.868548 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:05.868552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:05.868554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:05.868705 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:05.869124 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:05.869131 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:05.869393 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 12:38:05.869599 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 12:38:05.869948 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":153,"threads_started":1}
I20260619 12:38:05.871994 14621 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42511
I20260619 12:38:05.872268 14621 heartbeater.cc:464] Registering TS with master...
I20260619 12:38:05.872802 14621 heartbeater.cc:511] Master 127.0.98.62:42511 requested a full tablet report, sending...
I20260619 12:38:05.873921 14162 ts_manager.cc:194] Registered new tserver with Master: 869686f8889749978e8fb1bca3e3b729 (127.0.98.3:42635)
I20260619 12:38:05.874497 14162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59713
I20260619 12:38:05.878242 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:38:05.887668 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:05.880317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50360 (local address 127.0.98.62:42511)
0619 12:38:05.880457 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:05.880460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:05.880681 (+ 221us) server_negotiation.cc:407] Connection header received
0619 12:38:05.880833 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:05.880836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:05.880875 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:05.880945 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:05.881645 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.882152 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.882967 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.883171 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.883258 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:05.884632 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:05.884635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:05.884637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:05.884660 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:05.885998 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:05.886411 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:05.886414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:05.886415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:05.886494 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:05.886722 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:05.886724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:05.886725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:05.886947 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:05.887055 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:05.887339 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 12:38:05.887444 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 12:38:05.891031 14162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:38:05.894294 14162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:38:05.909777 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:05.903219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.0.98.1:39831)
0619 12:38:05.903641 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:05.903659 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:05.904860 (+ 1201us) server_negotiation.cc:407] Connection header received
0619 12:38:05.904947 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:05.904952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:05.905183 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:05.905329 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:05.905719 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.906442 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.908196 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.909035 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.909174 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:05.909336 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:05.909566 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 12:38:05.909651 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":59,"threads_started":1}
I20260619 12:38:05.909814 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:05.903037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39831 (local address 127.0.0.1:33170)
0619 12:38:05.904521 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:05.904534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:05.904620 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:05.905520 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:05.905522 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:05.905537 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:05.905608 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:05.905612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.906615 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.906618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:05.908074 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:05.908085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.909167 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.909192 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:05.909367 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:05.909415 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 12:38:05.909532 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 12:38:05.909665 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1399,"mutex_wait_us":1214,"thread_start_us":91,"threads_started":1}
I20260619 12:38:05.910315 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:05.903638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42635 (local address 127.0.0.1:54330)
0619 12:38:05.903896 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:05.903908 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:05.903990 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:05.905190 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:05.905195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:05.905214 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:05.905391 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:05.905399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.906776 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.906779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:05.908230 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:05.908241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.909232 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.909235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:05.909999 (+ 764us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:05.910025 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 12:38:05.910098 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 12:38:05.910166 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":681,"thread_start_us":113,"threads_started":1}
I20260619 12:38:05.910706 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:05.903037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37989 (local address 127.0.0.1:42254)
0619 12:38:05.903425 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:05.903506 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:05.903640 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:05.904980 (+ 1340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:05.904986 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:05.905011 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:05.905191 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:05.905202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.906731 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.906734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:05.908230 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:05.908241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.910318 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.910324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:05.910434 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:05.910456 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:38:05.910527 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 12:38:05.910591 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":226,"thread_start_us":114,"threads_started":1}
I20260619 12:38:05.910764 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:05.903907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.0.98.3:42635)
0619 12:38:05.904365 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:05.904376 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:05.904399 (+ 23us) server_negotiation.cc:407] Connection header received
0619 12:38:05.904474 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:05.904479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:05.904728 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:05.905018 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:05.905494 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.906629 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.908357 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.909124 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.909218 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:05.909336 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:05.910546 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0619 12:38:05.910653 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":103,"threads_started":1}
I20260619 12:38:05.911837 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:05.903830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.0.98.2:37989)
0619 12:38:05.904308 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:05.904322 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:05.904347 (+ 25us) server_negotiation.cc:407] Connection header received
0619 12:38:05.904432 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:05.904437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:05.904673 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:05.904975 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:05.905336 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.906593 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.909366 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:05.910198 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:05.910301 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:05.910452 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:05.911601 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0619 12:38:05.911701 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":90,"threads_started":1}
I20260619 12:38:05.913384 14287 tablet_service.cc:1467] Processing CreateTablet for tablet 194e9f4afc6f497398f1387fc8b422cb (DEFAULT_TABLE table=test-table [id=aee7cdc828ef4a2eb57c9967467e71aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:38:05.913563 14555 tablet_service.cc:1467] Processing CreateTablet for tablet 194e9f4afc6f497398f1387fc8b422cb (DEFAULT_TABLE table=test-table [id=aee7cdc828ef4a2eb57c9967467e71aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:38:05.914409 14287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194e9f4afc6f497398f1387fc8b422cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:05.914578 14555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194e9f4afc6f497398f1387fc8b422cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:05.914382 14421 tablet_service.cc:1467] Processing CreateTablet for tablet 194e9f4afc6f497398f1387fc8b422cb (DEFAULT_TABLE table=test-table [id=aee7cdc828ef4a2eb57c9967467e71aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:38:05.915304 14421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194e9f4afc6f497398f1387fc8b422cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:05.920728 14638 tablet_bootstrap.cc:492] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: Bootstrap starting.
I20260619 12:38:05.922600 14640 tablet_bootstrap.cc:492] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25: Bootstrap starting.
I20260619 12:38:05.923105 14638 tablet_bootstrap.cc:654] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:05.923828 14639 tablet_bootstrap.cc:492] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729: Bootstrap starting.
I20260619 12:38:05.923944 14638 log.cc:824] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:05.925209 14640 tablet_bootstrap.cc:654] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:05.926092 14638 tablet_bootstrap.cc:492] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: No bootstrap required, opened a new log
I20260619 12:38:05.926115 14639 tablet_bootstrap.cc:654] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:05.926304 14638 ts_tablet_manager.cc:1397] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:38:05.926282 14640 log.cc:824] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:05.927270 14639 log.cc:824] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:05.928499 14640 tablet_bootstrap.cc:492] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25: No bootstrap required, opened a new log
I20260619 12:38:05.928696 14640 ts_tablet_manager.cc:1397] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:38:05.929324 14639 tablet_bootstrap.cc:492] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729: No bootstrap required, opened a new log
I20260619 12:38:05.929505 14639 ts_tablet_manager.cc:1397] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:38:05.930326 14638 raft_consensus.cc:348] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } }
I20260619 12:38:05.930580 14638 raft_consensus.cc:374] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:05.930642 14638 raft_consensus.cc:729] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8ea6b60ebb44268af0f3c9ff0d90ac, State: Initialized, Role: FOLLOWER
I20260619 12:38:05.931102 14638 consensus_queue.cc:260] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } }
I20260619 12:38:05.931803 14353 heartbeater.cc:503] Master 127.0.98.62:42511 was elected leader, sending a full tablet report...
I20260619 12:38:05.932205 14638 ts_tablet_manager.cc:1428] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 12:38:05.932785 14640 raft_consensus.cc:348] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } }
I20260619 12:38:05.933143 14640 raft_consensus.cc:374] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:05.933197 14640 raft_consensus.cc:729] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da0a02cfb2b04eb685bd7503ff955c25, State: Initialized, Role: FOLLOWER
I20260619 12:38:05.933342 14639 raft_consensus.cc:348] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } }
I20260619 12:38:05.933625 14639 raft_consensus.cc:374] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:05.933686 14639 raft_consensus.cc:729] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869686f8889749978e8fb1bca3e3b729, State: Initialized, Role: FOLLOWER
I20260619 12:38:05.933897 14640 consensus_queue.cc:260] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } }
I20260619 12:38:05.934054 14639 consensus_queue.cc:260] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } }
I20260619 12:38:05.934620 14487 heartbeater.cc:503] Master 127.0.98.62:42511 was elected leader, sending a full tablet report...
I20260619 12:38:05.934924 14639 ts_tablet_manager.cc:1428] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260619 12:38:05.934924 14640 ts_tablet_manager.cc:1428] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260619 12:38:05.935338 14621 heartbeater.cc:503] Master 127.0.98.62:42511 was elected leader, sending a full tablet report...
I20260619 12:38:06.073141 14644 raft_consensus.cc:482] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:38:06.073454 14644 raft_consensus.cc:504] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } }
I20260619 12:38:06.074705 14644 leader_election.cc:288] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da0a02cfb2b04eb685bd7503ff955c25 (127.0.98.2:37989), 869686f8889749978e8fb1bca3e3b729 (127.0.98.3:42635)
I20260619 12:38:06.078858 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:06.075080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49817 (local address 127.0.98.3:42635)
0619 12:38:06.075278 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:06.075283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:06.075539 (+ 256us) server_negotiation.cc:407] Connection header received
0619 12:38:06.075798 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:06.075801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:06.075846 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:06.075943 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:06.076249 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.076713 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.077828 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.078317 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.078434 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:06.078499 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:06.078664 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 12:38:06.078739 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 12:38:06.079032 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:06.075098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47721 (local address 127.0.98.2:37989)
0619 12:38:06.075252 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:06.075257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:06.075634 (+ 377us) server_negotiation.cc:407] Connection header received
0619 12:38:06.075765 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:06.075772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:06.075816 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:06.075875 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:06.076189 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.076706 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.077829 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.078393 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.078491 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:06.078552 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:06.078809 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 12:38:06.078882 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:38:06.078904 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:06.074995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42635 (local address 127.0.98.1:49817)
0619 12:38:06.075417 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:06.075429 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:06.075644 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:06.075988 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:06.075991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:06.076006 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:06.076114 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:06.076118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.076847 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.076850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:06.077706 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:06.077714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.078442 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.078446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:06.078550 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:06.078573 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:38:06.078648 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 12:38:06.078724 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":28,"thread_start_us":162,"threads_started":1}
I20260619 12:38:06.079416 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:06.074997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37989 (local address 127.0.98.1:47721)
0619 12:38:06.075543 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:06.075555 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:06.075656 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:06.075909 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:06.075914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:06.075945 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:06.076080 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:06.076087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.076815 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.076818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:06.077703 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:06.077712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.078569 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.078572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:06.078669 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:06.078687 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 12:38:06.079203 (+ 516us) client_negotiation.cc:240] Negotiation successful
0619 12:38:06.079280 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":32,"spinlock_wait_cycles":4864,"thread_start_us":148,"threads_started":1}
I20260619 12:38:06.079962 14575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194e9f4afc6f497398f1387fc8b422cb" candidate_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869686f8889749978e8fb1bca3e3b729" is_pre_election: true
I20260619 12:38:06.080065 14441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194e9f4afc6f497398f1387fc8b422cb" candidate_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da0a02cfb2b04eb685bd7503ff955c25" is_pre_election: true
I20260619 12:38:06.080283 14575 raft_consensus.cc:2454] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8ea6b60ebb44268af0f3c9ff0d90ac in term 0.
I20260619 12:38:06.080353 14441 raft_consensus.cc:2454] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8ea6b60ebb44268af0f3c9ff0d90ac in term 0.
I20260619 12:38:06.080796 14243 leader_election.cc:302] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 869686f8889749978e8fb1bca3e3b729, da0a02cfb2b04eb685bd7503ff955c25, fe8ea6b60ebb44268af0f3c9ff0d90ac; no voters:
I20260619 12:38:06.081091 14644 raft_consensus.cc:2781] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:38:06.081182 14644 raft_consensus.cc:482] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:38:06.081219 14644 raft_consensus.cc:3037] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:06.082485 14644 raft_consensus.cc:504] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } }
I20260619 12:38:06.082858 14644 leader_election.cc:288] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [CANDIDATE]: Term 1 election: Requested vote from peers da0a02cfb2b04eb685bd7503ff955c25 (127.0.98.2:37989), 869686f8889749978e8fb1bca3e3b729 (127.0.98.3:42635)
I20260619 12:38:06.083218 14441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194e9f4afc6f497398f1387fc8b422cb" candidate_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da0a02cfb2b04eb685bd7503ff955c25"
I20260619 12:38:06.083290 14575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194e9f4afc6f497398f1387fc8b422cb" candidate_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869686f8889749978e8fb1bca3e3b729"
I20260619 12:38:06.083343 14441 raft_consensus.cc:3037] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:06.083436 14575 raft_consensus.cc:3037] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:06.084518 14575 raft_consensus.cc:2454] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8ea6b60ebb44268af0f3c9ff0d90ac in term 1.
I20260619 12:38:06.084693 14441 raft_consensus.cc:2454] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8ea6b60ebb44268af0f3c9ff0d90ac in term 1.
I20260619 12:38:06.084828 14243 leader_election.cc:302] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869686f8889749978e8fb1bca3e3b729, fe8ea6b60ebb44268af0f3c9ff0d90ac; no voters:
I20260619 12:38:06.085114 14644 raft_consensus.cc:2781] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:38:06.085345 14644 raft_consensus.cc:686] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 LEADER]: Becoming Leader. State: Replica: fe8ea6b60ebb44268af0f3c9ff0d90ac, State: Running, Role: LEADER
I20260619 12:38:06.085685 14644 consensus_queue.cc:237] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } }
I20260619 12:38:06.089272 14162 catalog_manager.cc:5495] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac reported cstate change: term changed from 0 to 1, leader changed from <none> to fe8ea6b60ebb44268af0f3c9ff0d90ac (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } health_report { overall_health: UNKNOWN } } }
W20260619 12:38:06.106213 14622 tablet.cc:2369] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:38:06.131300 14488 tablet.cc:2369] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:38:06.145722 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:06.142105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.0.98.1:39831)
0619 12:38:06.142245 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:06.142249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:06.142262 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:38:06.142340 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:06.142342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:06.142398 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:06.142489 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:38:06.142703 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.143203 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.144181 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.144402 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.144495 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:06.144809 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:38:06.145262 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:38:06.145363 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:06.145441 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 12:38:06.145497 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 12:38:06.150267 14354 tablet.cc:2369] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:38:06.152661 14575 raft_consensus.cc:1261] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729 [term 1 FOLLOWER]: Refusing update from remote peer fe8ea6b60ebb44268af0f3c9ff0d90ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:38:06.152661 14441 raft_consensus.cc:1261] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25 [term 1 FOLLOWER]: Refusing update from remote peer fe8ea6b60ebb44268af0f3c9ff0d90ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:38:06.153537 14644 consensus_queue.cc:1035] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:38:06.153817 14649 consensus_queue.cc:1035] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:38:06.163679 14653 mvcc.cc:205] Tried to move back new op lower bound from 7298550522471542784 to 7298550522209693696. Current Snapshot: MvccSnapshot[applied={T|T < 7298550522471542784}]
I20260619 12:38:06.170174 14161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:38:06.170392 14161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:38:06.173080 14161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc: Sending DeleteTablet for 3 replicas of tablet 194e9f4afc6f497398f1387fc8b422cb
I20260619 12:38:06.174012 14555 tablet_service.cc:1514] Processing DeleteTablet for tablet 194e9f4afc6f497398f1387fc8b422cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:38:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54330
I20260619 12:38:06.174012 14287 tablet_service.cc:1514] Processing DeleteTablet for tablet 194e9f4afc6f497398f1387fc8b422cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:38:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33170
I20260619 12:38:06.174222 14421 tablet_service.cc:1514] Processing DeleteTablet for tablet 194e9f4afc6f497398f1387fc8b422cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:38:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42254
I20260619 12:38:06.174562 14662 tablet_replica.cc:330] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: stopping tablet replica
I20260619 12:38:06.174777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 14224
I20260619 12:38:06.174947 14662 raft_consensus.cc:2229] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:38:06.174811 14663 tablet_replica.cc:330] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25: stopping tablet replica
I20260619 12:38:06.174984 14661 tablet_replica.cc:330] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729: stopping tablet replica
I20260619 12:38:06.175766 14661 raft_consensus.cc:2229] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:38:06.176137 14663 raft_consensus.cc:2229] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:38:06.176566 14661 raft_consensus.cc:2258] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:38:06.176910 14663 raft_consensus.cc:2258] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:38:06.178583 14663 ts_tablet_manager.cc:1905] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:38:06.178637 14661 ts_tablet_manager.cc:1905] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:38:06.181003 14663 ts_tablet_manager.cc:1918] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:38:06.181161 14663 log.cc:1196] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/194e9f4afc6f497398f1387fc8b422cb
I20260619 12:38:06.181497 14663 ts_tablet_manager.cc:1939] T 194e9f4afc6f497398f1387fc8b422cb P da0a02cfb2b04eb685bd7503ff955c25: Deleting consensus metadata
I20260619 12:38:06.182209 14661 ts_tablet_manager.cc:1918] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:38:06.182327 14661 log.cc:1196] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/194e9f4afc6f497398f1387fc8b422cb
I20260619 12:38:06.182653 14661 ts_tablet_manager.cc:1939] T 194e9f4afc6f497398f1387fc8b422cb P 869686f8889749978e8fb1bca3e3b729: Deleting consensus metadata
I20260619 12:38:06.182862 14146 catalog_manager.cc:4841] TS da0a02cfb2b04eb685bd7503ff955c25 (127.0.98.2:37989): tablet 194e9f4afc6f497398f1387fc8b422cb (table test-table [id=aee7cdc828ef4a2eb57c9967467e71aa]) successfully deleted
W20260619 12:38:06.183287 14147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39831 (error 108)
I20260619 12:38:06.183547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 14358
I20260619 12:38:06.185166 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:06.184859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39831 (local address 127.0.0.1:33192)
0619 12:38:06.184993 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:06.185073 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 12:38:06.185770 14147 catalog_manager.cc:4584] TS fe8ea6b60ebb44268af0f3c9ff0d90ac (127.0.98.1:39831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 194e9f4afc6f497398f1387fc8b422cb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39831: connect: Connection refused (error 111)
I20260619 12:38:06.185861 14149 catalog_manager.cc:4841] TS 869686f8889749978e8fb1bca3e3b729 (127.0.98.3:42635): tablet 194e9f4afc6f497398f1387fc8b422cb (table test-table [id=aee7cdc828ef4a2eb57c9967467e71aa]) successfully deleted
I20260619 12:38:06.189447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 14492
I20260619 12:38:06.196031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 14130
I20260619 12:38:06.202939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42511
--webserver_interface=127.0.98.62
--webserver_port=39205
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:38:06.315022 14665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:06.315316 14665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:06.315358 14665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:06.319451 14665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:38:06.319536 14665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:06.319556 14665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:06.319573 14665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:38:06.319638 14665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:38:06.325021 14665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:06.326224 14665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:06.327330 14665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:06.341408 14671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:06.341439 14672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:06.341442 14674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:06.341555 14665 server_base.cc:1034] running on GCE node
I20260619 12:38:06.342460 14665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:06.343915 14665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:06.345139 14665 hybrid_clock.cc:648] HybridClock initialized: now 1781872686345093 us; error 63 us; skew 500 ppm
Jun 19 12:38:06 dist-test-slave-kdrb krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:06.348404 14665 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:38:06.349587 14665 webserver.cc:460] Webserver started at http://127.0.98.62:39205/ using document root <none> and password file <none>
I20260619 12:38:06.350260 14665 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:06.350387 14665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:06.355700 14665 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:38:06.358402 14681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:06.359948 14665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:38:06.360122 14665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "69c5c649c01847158f9a7230870bd4bc"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "6f26face1203a3ab3a909f799f4a5ddb"
server_key_iv: "1cfa330974990f42586ef0e6987837ef"
server_key_version: "encryptionkey@0"
I20260619 12:38:06.360827 14665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:06.391970 14665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:06.395660 14665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:06.395915 14665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:06.403702 14665 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42511
I20260619 12:38:06.404013 14733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42511 every 8 connection(s)
I20260619 12:38:06.404918 14665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:38:06.409559 14734 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:38:06.410034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 14665
I20260619 12:38:06.410477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39831
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42511
--webserver_port=42059
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:38:06.411569 14734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc: Bootstrap starting.
I20260619 12:38:06.429610 14734 log.cc:824] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:06.438670 14734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:38:06.439127 14734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc: Bootstrap complete.
I20260619 12:38:06.443346 14734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } }
I20260619 12:38:06.443852 14734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c5c649c01847158f9a7230870bd4bc, State: Initialized, Role: FOLLOWER
I20260619 12:38:06.444383 14734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } }
I20260619 12:38:06.444500 14734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:38:06.444592 14734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:38:06.444711 14734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:38:06.446766 14734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } }
I20260619 12:38:06.447140 14734 leader_election.cc:302] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69c5c649c01847158f9a7230870bd4bc; no voters:
I20260619 12:38:06.447518 14734 leader_election.cc:288] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 12:38:06.447635 14738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 2 FOLLOWER]: Leader election won for term 2
I20260619 12:38:06.447841 14738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 2 LEADER]: Becoming Leader. State: Replica: 69c5c649c01847158f9a7230870bd4bc, State: Running, Role: LEADER
I20260619 12:38:06.448151 14738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } }
I20260619 12:38:06.448895 14734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:38:06.449159 14738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69c5c649c01847158f9a7230870bd4bc. Latest consensus state: current_term: 2 leader_uuid: "69c5c649c01847158f9a7230870bd4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } }
I20260619 12:38:06.449287 14738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [sys.catalog]: This master's current role is: LEADER
I20260619 12:38:06.449941 14739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "69c5c649c01847158f9a7230870bd4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } }
I20260619 12:38:06.450114 14739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [sys.catalog]: This master's current role is: LEADER
I20260619 12:38:06.450158 14742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:38:06.455235 14742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:38:06.455899 14742 catalog_manager.cc:1259] Loaded cluster ID: 1978d65ee6fe40d0bf0abeaea1ef3c82
I20260619 12:38:06.455987 14742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:38:06.457414 14742 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 12:38:06.462278 14742 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 12:38:06.470369 14742 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:38:06.471328 14742 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc: Loaded TSK: 0
I20260619 12:38:06.472699 14742 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:38:06.534164 14736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:06.534498 14736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:06.534579 14736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:06.539322 14736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:06.539435 14736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:06.539553 14736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:38:06.547720 14736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42059
--tserver_master_addrs=127.0.98.62:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:06.549211 14736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:06.550608 14736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:06.561853 14762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:06.561854 14764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:06.561853 14761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:06.562296 14736 server_base.cc:1034] running on GCE node
I20260619 12:38:06.562726 14736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:06.563526 14736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:06.564733 14736 hybrid_clock.cc:648] HybridClock initialized: now 1781872686564715 us; error 36 us; skew 500 ppm
Jun 19 12:38:06 dist-test-slave-kdrb krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:06.567907 14736 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:38:06.568997 14736 webserver.cc:460] Webserver started at http://127.0.98.1:42059/ using document root <none> and password file <none>
I20260619 12:38:06.569589 14736 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:06.569664 14736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:06.573904 14736 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 12:38:06.576278 14771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:06.577669 14736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 12:38:06.577802 14736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "5528c98868d005bec62fb8bd75f80dac"
server_key_iv: "d19762f5e07ab6c00848edf20c0a4a8d"
server_key_version: "encryptionkey@0"
I20260619 12:38:06.578433 14736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:06.597499 14736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:06.600991 14736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:06.601224 14736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:06.601869 14736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:38:06.602941 14778 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 12:38:06.606988 14736 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 12:38:06.607067 14736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 12:38:06.607153 14736 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 12:38:06.609001 14736 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 12:38:06.609054 14736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 12:38:06.609385 14778 tablet_bootstrap.cc:492] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: Bootstrap starting.
I20260619 12:38:06.620435 14736 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39831
I20260619 12:38:06.620512 14885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39831 every 8 connection(s)
I20260619 12:38:06.621554 14736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:38:06.628226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 14736
I20260619 12:38:06.628731 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37989
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42511
--webserver_port=43471
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:38:06.629164 14778 log.cc:824] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:38:06 dist-test-slave-kdrb krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:38:06.639533 14778 tablet_bootstrap.cc:492] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:38:06.640035 14778 tablet_bootstrap.cc:492] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: Bootstrap complete.
I20260619 12:38:06.640784 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:06.624035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:56003)
0619 12:38:06.624670 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:06.624744 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:06.625994 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:06.627354 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:06.627364 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:06.627946 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:06.628620 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:06.628648 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.630711 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.630714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:06.631716 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:06.631728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.632572 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.632576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:06.632772 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:06.633442 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:06.633459 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:06.635096 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:06.637454 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:06.637460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:06.637474 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:06.637761 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:06.638152 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:06.638155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:06.638157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:06.638308 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:06.638933 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:06.638939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:06.639222 (+ 283us) client_negotiation.cc:769] Sending connection context
0619 12:38:06.639421 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 12:38:06.639832 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":153,"threads_started":1}
I20260619 12:38:06.640771 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:06.624044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56003 (local address 127.0.98.62:42511)
0619 12:38:06.624470 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:06.624491 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:06.624885 (+ 394us) server_negotiation.cc:407] Connection header received
0619 12:38:06.626509 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:06.626527 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:06.626880 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:06.627203 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:06.628888 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.630505 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.631867 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.632392 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.632615 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:06.635322 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:06.635328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:06.635341 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:06.635374 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:06.637306 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:06.637903 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:06.637908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:06.637913 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:06.638036 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:06.638419 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:06.638429 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:06.638430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:06.638802 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:06.638951 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:06.639588 (+ 637us) server_negotiation.cc:299] Negotiation successful
0619 12:38:06.639990 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":137,"threads_started":1}
I20260619 12:38:06.641592 14778 ts_tablet_manager.cc:1397] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.012s
I20260619 12:38:06.643787 14886 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42511
I20260619 12:38:06.644049 14886 heartbeater.cc:464] Registering TS with master...
I20260619 12:38:06.644567 14886 heartbeater.cc:511] Master 127.0.98.62:42511 requested a full tablet report, sending...
I20260619 12:38:06.644655 14778 raft_consensus.cc:348] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } }
I20260619 12:38:06.645184 14778 raft_consensus.cc:729] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8ea6b60ebb44268af0f3c9ff0d90ac, State: Initialized, Role: FOLLOWER
I20260619 12:38:06.645702 14778 consensus_queue.cc:260] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } }
I20260619 12:38:06.646746 14778 ts_tablet_manager.cc:1428] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 12:38:06.649037 14698 ts_manager.cc:194] Registered new tserver with Master: fe8ea6b60ebb44268af0f3c9ff0d90ac (127.0.98.1:39831)
I20260619 12:38:06.653029 14698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56003
I20260619 12:38:06.655989 14886 heartbeater.cc:503] Master 127.0.98.62:42511 was elected leader, sending a full tablet report...
Jun 19 12:38:06 dist-test-slave-kdrb krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:38:06.664819 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:06.652484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39831 (local address 127.0.0.1:33208)
0619 12:38:06.652958 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:06.653009 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:06.653185 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:06.654195 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:06.654201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:06.654647 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:06.654784 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:06.654792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.656292 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.656295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:06.657209 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:06.657230 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.658046 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.658064 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:06.658180 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:06.658823 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:06.658839 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:06.660497 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:06.662640 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:06.662652 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:06.662658 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:06.662916 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:06.663256 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:06.663259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:06.663261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:06.663352 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:06.664031 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:06.664049 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:06.664218 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 12:38:06.664386 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 12:38:06.664624 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":142,"threads_started":1}
I20260619 12:38:06.664821 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:06.653293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.0.98.1:39831)
0619 12:38:06.653619 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:06.653633 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:06.653659 (+ 26us) server_negotiation.cc:407] Connection header received
0619 12:38:06.653743 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:06.653748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:06.653983 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:06.654179 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:06.654931 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.656111 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.657362 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.657944 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.658038 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:06.660702 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:06.660711 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:06.660722 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:06.660747 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:06.662496 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:06.663056 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:06.663061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:06.663063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:06.663148 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:06.663447 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:06.663452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:06.663454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:06.663799 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:06.664057 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:06.664486 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 12:38:06.664676 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":115,"threads_started":1}
I20260619 12:38:06.665954 14820 tablet_service.cc:1514] Processing DeleteTablet for tablet 194e9f4afc6f497398f1387fc8b422cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:38:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33208
I20260619 12:38:06.666823 14895 tablet_replica.cc:330] stopping tablet replica
I20260619 12:38:06.667166 14895 raft_consensus.cc:2229] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:38:06.667447 14895 raft_consensus.cc:2258] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:38:06.668462 14895 ts_tablet_manager.cc:1905] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:38:06.672151 14895 ts_tablet_manager.cc:1918] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:38:06.672251 14895 log.cc:1196] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal/wals/194e9f4afc6f497398f1387fc8b422cb
I20260619 12:38:06.672546 14895 ts_tablet_manager.cc:1939] T 194e9f4afc6f497398f1387fc8b422cb P fe8ea6b60ebb44268af0f3c9ff0d90ac: Deleting consensus metadata
I20260619 12:38:06.674362 14683 catalog_manager.cc:4841] TS fe8ea6b60ebb44268af0f3c9ff0d90ac (127.0.98.1:39831): tablet 194e9f4afc6f497398f1387fc8b422cb (table test-table [id=aee7cdc828ef4a2eb57c9967467e71aa]) successfully deleted
W20260619 12:38:06.749405 14891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:06.749727 14891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:06.749823 14891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:06.754137 14891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:06.754251 14891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:06.754346 14891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:38:06.759929 14891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43471
--tserver_master_addrs=127.0.98.62:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:06.761165 14891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:06.762324 14891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:06.772933 14901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:06.772979 14900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:06.772979 14903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:06.773094 14891 server_base.cc:1034] running on GCE node
I20260619 12:38:06.773705 14891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:06.774355 14891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:06.775548 14891 hybrid_clock.cc:648] HybridClock initialized: now 1781872686775532 us; error 32 us; skew 500 ppm
Jun 19 12:38:06 dist-test-slave-kdrb krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:06.778307 14891 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:38:06.779345 14891 webserver.cc:460] Webserver started at http://127.0.98.2:43471/ using document root <none> and password file <none>
I20260619 12:38:06.780045 14891 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:06.780161 14891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:06.785218 14891 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 12:38:06.787446 14910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:06.788821 14891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:38:06.788976 14891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "da0a02cfb2b04eb685bd7503ff955c25"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "b7ad49f910b323a226f629ea7481103c"
server_key_iv: "e7de355d38f42f4737a69c9821de6e03"
server_key_version: "encryptionkey@0"
I20260619 12:38:06.789572 14891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:06.806280 14891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:06.809599 14891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:06.809846 14891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:06.810513 14891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:38:06.811385 14891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:38:06.811461 14891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:06.811547 14891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:38:06.811592 14891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:06.820672 14891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37989
I20260619 12:38:06.820709 15023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37989 every 8 connection(s)
I20260619 12:38:06.821779 14891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:38:06.826349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 14891
I20260619 12:38:06.826809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42635
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42511
--webserver_port=42989
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:38:06 dist-test-slave-kdrb krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:38:06.836892 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:06.823744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58251 (local address 127.0.98.62:42511)
0619 12:38:06.823887 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:06.823891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:06.824784 (+ 893us) server_negotiation.cc:407] Connection header received
0619 12:38:06.826146 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:06.826149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:06.826215 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:06.826325 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:06.827791 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.828339 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.829563 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.829814 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.829934 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:06.832900 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:06.832903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:06.832905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:06.832936 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:06.834455 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:06.835043 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:06.835047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:06.835050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:06.835195 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:06.835682 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:06.835686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:06.835688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:06.835894 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:06.836007 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:06.836567 (+ 560us) server_negotiation.cc:299] Negotiation successful
0619 12:38:06.836717 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:38:06.837805 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:06.824181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:58251)
0619 12:38:06.824633 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:06.824676 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:06.825911 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:06.826542 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:06.826555 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:06.826996 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:06.827620 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:06.827637 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.828488 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.828496 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:06.829420 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:06.829431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:06.829981 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:06.829985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:06.830166 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:06.830856 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:06.830874 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:06.832700 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:06.834590 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:06.834596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:06.834615 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:06.834908 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:06.835317 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:06.835320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:06.835322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:06.835580 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:06.836007 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:06.836016 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:06.836311 (+ 295us) client_negotiation.cc:769] Sending connection context
0619 12:38:06.836513 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 12:38:06.836887 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":119,"threads_started":1}
I20260619 12:38:06.839134 15024 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42511
I20260619 12:38:06.839403 15024 heartbeater.cc:464] Registering TS with master...
I20260619 12:38:06.839955 15024 heartbeater.cc:511] Master 127.0.98.62:42511 requested a full tablet report, sending...
I20260619 12:38:06.841051 14698 ts_manager.cc:194] Registered new tserver with Master: da0a02cfb2b04eb685bd7503ff955c25 (127.0.98.2:37989)
I20260619 12:38:06.841646 14698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58251
W20260619 12:38:06.944799 15027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:06.945139 15027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:06.945235 15027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:06.949499 15027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:06.949595 15027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:06.949682 15027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:38:06.955554 15027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42989
--tserver_master_addrs=127.0.98.62:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:06.956727 15027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:06.957818 15027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:38:06.968117 15027 server_base.cc:1034] running on GCE node
W20260619 12:38:06.968078 15033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:06.968063 15035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:06.968057 15032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:06.968928 15027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:06.969614 15027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:06.970822 15027 hybrid_clock.cc:648] HybridClock initialized: now 1781872686970803 us; error 33 us; skew 500 ppm
Jun 19 12:38:06 dist-test-slave-kdrb krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:06.973935 15027 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:38:06.974890 15027 webserver.cc:460] Webserver started at http://127.0.98.3:42989/ using document root <none> and password file <none>
I20260619 12:38:06.975553 15027 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:06.975639 15027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:06.979763 15027 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:38:06.982002 15042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:06.983639 15027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:38:06.983772 15027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "869686f8889749978e8fb1bca3e3b729"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "701a7555e02c7e40eeb973f5e023011a"
server_key_iv: "3f5df402b22e936c5e082756ace7db9c"
server_key_version: "encryptionkey@0"
I20260619 12:38:06.984365 15027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:06.996169 15027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:06.999370 15027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:06.999631 15027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:07.000238 15027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:38:07.001058 15027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:38:07.001106 15027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:07.001175 15027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:38:07.001216 15027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:07.009984 15027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42635
I20260619 12:38:07.010006 15155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42635 every 8 connection(s)
I20260619 12:38:07.011068 15027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:38:07.013666 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 15027
Jun 19 12:38:07 dist-test-slave-kdrb krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:38:07.023643 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.014360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.0.98.62:42511)
0619 12:38:07.015048 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:07.015053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:07.015065 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:38:07.015130 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:07.015135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:07.015180 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:07.015276 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:07.016165 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.016939 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.017977 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.018257 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.018386 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:07.019905 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:07.019908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:07.019910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:07.019934 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:07.021741 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:07.022201 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:07.022204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:07.022206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:07.022302 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:07.022597 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:07.022601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:07.022603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:07.022789 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:07.023008 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:07.023194 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 12:38:07.023374 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":75,"threads_started":1}
I20260619 12:38:07.026549 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.013192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57937 (local address 127.0.98.62:42511)
0619 12:38:07.013405 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:07.013411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:07.014880 (+ 1469us) server_negotiation.cc:407] Connection header received
0619 12:38:07.015940 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:07.015943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:07.015991 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:07.016092 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:07.017556 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.018185 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.019703 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.019990 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.020069 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:07.022842 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:07.022847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:07.022855 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:07.022878 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:07.024528 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:07.025059 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:07.025064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:07.025071 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:07.025140 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:07.025493 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:07.025498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:07.025499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:07.025669 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:07.025784 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:07.026289 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 12:38:07.026403 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 12:38:07.027454 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.013623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:57937)
0619 12:38:07.014705 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:07.014769 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:07.015736 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:07.016310 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:07.016320 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:07.016826 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:07.017416 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:07.017430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.018326 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.018331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:07.019579 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:07.019589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.020157 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.020163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:07.020377 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:07.020918 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:07.020962 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:07.022660 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:07.024647 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:07.024653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:07.024670 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:07.024942 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:07.025244 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:07.025247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:07.025249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:07.025395 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:07.025784 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:07.025793 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:07.026052 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 12:38:07.026251 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 12:38:07.026605 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":657,"threads_started":1}
I20260619 12:38:07.028697 15156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42511
I20260619 12:38:07.028970 15156 heartbeater.cc:464] Registering TS with master...
I20260619 12:38:07.029493 15156 heartbeater.cc:511] Master 127.0.98.62:42511 requested a full tablet report, sending...
I20260619 12:38:07.030537 14698 ts_manager.cc:194] Registered new tserver with Master: 869686f8889749978e8fb1bca3e3b729 (127.0.98.3:42635)
I20260619 12:38:07.031139 14698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57937
I20260619 12:38:07.037283 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:38:07.041446 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.037906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.0.98.62:42511)
0619 12:38:07.038041 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:07.038045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:07.038261 (+ 216us) server_negotiation.cc:407] Connection header received
0619 12:38:07.038359 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:07.038361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:07.038465 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:07.038555 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:38:07.038741 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.039229 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.040117 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.040329 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.040408 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:07.040672 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:38:07.041049 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:38:07.041136 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:07.041223 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 12:38:07.041290 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:38:07.042078 14698 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:38:07.045285 14698 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:38:07.053486 14820 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca7fa19bbee4c8e9cab71ac43898acf (DEFAULT_TABLE table=test-table [id=bf8d93cf811045aa93e0df2a8005ed2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:38:07.054193 14820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca7fa19bbee4c8e9cab71ac43898acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:07.057142 14778 tablet_bootstrap.cc:492] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac: Bootstrap starting.
I20260619 12:38:07.057914 14778 tablet_bootstrap.cc:654] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:07.058902 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.052295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37989 (local address 127.0.0.1:42264)
0619 12:38:07.052434 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:07.052452 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:07.052537 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:07.054394 (+ 1857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:07.054397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:07.054415 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:07.054543 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:07.054548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.055897 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.055900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:07.057145 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:07.057155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.058373 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.058378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:07.058498 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:07.058551 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 12:38:07.058639 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 12:38:07.058733 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":32}
I20260619 12:38:07.059018 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.052630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42635 (local address 127.0.0.1:54336)
0619 12:38:07.053499 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:07.053516 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:07.053661 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:07.054383 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:07.054386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:07.054405 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:07.054533 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:07.054540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.055650 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.055655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:07.057153 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:07.057159 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.058548 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.058551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:07.058636 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:07.058665 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 12:38:07.058758 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 12:38:07.058834 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":281,"thread_start_us":110,"threads_started":1}
I20260619 12:38:07.059041 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.053134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.0.98.3:42635)
0619 12:38:07.053430 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:07.053439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:07.053622 (+ 183us) server_negotiation.cc:407] Connection header received
0619 12:38:07.053920 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:07.053929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:07.054138 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:07.054315 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:07.054663 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.055419 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.057467 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.058225 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.058329 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:07.058479 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:07.058798 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 12:38:07.058897 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":104,"threads_started":1}
I20260619 12:38:07.059587 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.052511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.0.98.2:37989)
0619 12:38:07.053927 (+ 1416us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:07.053938 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:07.053960 (+ 22us) server_negotiation.cc:407] Connection header received
0619 12:38:07.054033 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:07.054038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:07.054237 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:07.054505 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:07.054677 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.055756 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.057388 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.058236 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.058334 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:07.058479 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:07.059280 (+ 801us) server_negotiation.cc:299] Negotiation successful
0619 12:38:07.059379 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1306,"thread_start_us":189,"threads_started":1}
I20260619 12:38:07.059958 14778 tablet_bootstrap.cc:492] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac: No bootstrap required, opened a new log
I20260619 12:38:07.060055 14778 ts_tablet_manager.cc:1397] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 12:38:07.060497 14778 raft_consensus.cc:348] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:07.060614 14778 raft_consensus.cc:374] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:07.060654 14778 raft_consensus.cc:729] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8ea6b60ebb44268af0f3c9ff0d90ac, State: Initialized, Role: FOLLOWER
I20260619 12:38:07.060779 14778 consensus_queue.cc:260] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:07.061026 14778 ts_tablet_manager.cc:1428] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:38:07.062282 14958 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca7fa19bbee4c8e9cab71ac43898acf (DEFAULT_TABLE table=test-table [id=bf8d93cf811045aa93e0df2a8005ed2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:38:07.062270 15090 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca7fa19bbee4c8e9cab71ac43898acf (DEFAULT_TABLE table=test-table [id=bf8d93cf811045aa93e0df2a8005ed2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:38:07.063313 15090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca7fa19bbee4c8e9cab71ac43898acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:07.063304 14958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca7fa19bbee4c8e9cab71ac43898acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:07.070472 15170 tablet_bootstrap.cc:492] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: Bootstrap starting.
I20260619 12:38:07.070528 15169 tablet_bootstrap.cc:492] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25: Bootstrap starting.
I20260619 12:38:07.072728 15170 tablet_bootstrap.cc:654] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:07.073199 15169 tablet_bootstrap.cc:654] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:07.074000 15170 log.cc:824] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:07.073999 15169 log.cc:824] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:07.076251 15169 tablet_bootstrap.cc:492] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25: No bootstrap required, opened a new log
I20260619 12:38:07.076465 15169 ts_tablet_manager.cc:1397] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:38:07.076854 15170 tablet_bootstrap.cc:492] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: No bootstrap required, opened a new log
I20260619 12:38:07.077042 15170 ts_tablet_manager.cc:1397] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 12:38:07.080603 15169 raft_consensus.cc:348] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:07.080766 15170 raft_consensus.cc:348] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:07.080940 15169 raft_consensus.cc:374] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:07.080992 15169 raft_consensus.cc:729] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da0a02cfb2b04eb685bd7503ff955c25, State: Initialized, Role: FOLLOWER
I20260619 12:38:07.080999 15170 raft_consensus.cc:374] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:07.081054 15170 raft_consensus.cc:729] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869686f8889749978e8fb1bca3e3b729, State: Initialized, Role: FOLLOWER
I20260619 12:38:07.081506 15170 consensus_queue.cc:260] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:07.081506 15169 consensus_queue.cc:260] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:07.082384 15156 heartbeater.cc:503] Master 127.0.98.62:42511 was elected leader, sending a full tablet report...
I20260619 12:38:07.082628 15170 ts_tablet_manager.cc:1428] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:38:07.082615 15024 heartbeater.cc:503] Master 127.0.98.62:42511 was elected leader, sending a full tablet report...
I20260619 12:38:07.082628 15169 ts_tablet_manager.cc:1428] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:38:07.084707 15174 raft_consensus.cc:482] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:38:07.084860 15174 raft_consensus.cc:504] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:07.086591 15174 leader_election.cc:288] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da0a02cfb2b04eb685bd7503ff955c25 (127.0.98.2:37989), fe8ea6b60ebb44268af0f3c9ff0d90ac (127.0.98.1:39831)
I20260619 12:38:07.090672 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.086701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41349 (local address 127.0.98.2:37989)
0619 12:38:07.087107 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:07.087114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:07.087132 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:38:07.087184 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:07.087187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:07.087243 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:07.087323 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:07.087599 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.088290 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.089498 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.090146 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.090266 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:07.090341 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:07.090479 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 12:38:07.090552 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331}
I20260619 12:38:07.090672 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.086298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37989 (local address 127.0.98.3:41349)
0619 12:38:07.086829 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:07.086846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:07.086941 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:07.087344 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:07.087346 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:07.087362 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:07.087459 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:07.087463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.088422 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.088426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:07.089364 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:07.089373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.090269 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.090271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:07.090351 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:07.090370 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 12:38:07.090460 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 12:38:07.090535 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444}
I20260619 12:38:07.091488 14978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca7fa19bbee4c8e9cab71ac43898acf" candidate_uuid: "869686f8889749978e8fb1bca3e3b729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da0a02cfb2b04eb685bd7503ff955c25" is_pre_election: true
I20260619 12:38:07.091691 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.086350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39831 (local address 127.0.98.3:42109)
0619 12:38:07.087092 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:07.087107 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:07.087189 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:07.088042 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:07.088045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:07.088064 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:07.088208 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:07.088220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.089181 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.089183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:07.090317 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:07.090327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.091234 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.091240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:07.091337 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:07.091359 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:38:07.091447 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 12:38:07.091556 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":118,"threads_started":1}
I20260619 12:38:07.091775 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.086845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42109 (local address 127.0.98.1:39831)
0619 12:38:07.087740 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:07.087746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:07.087764 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:38:07.087823 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:07.087826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:07.087892 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:07.088005 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:07.088367 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.089044 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.090428 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.091067 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.091193 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:07.091369 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:07.091460 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 12:38:07.091605 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561}
I20260619 12:38:07.091868 14978 raft_consensus.cc:2454] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869686f8889749978e8fb1bca3e3b729 in term 0.
I20260619 12:38:07.092281 14840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca7fa19bbee4c8e9cab71ac43898acf" candidate_uuid: "869686f8889749978e8fb1bca3e3b729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" is_pre_election: true
I20260619 12:38:07.092402 15043 leader_election.cc:302] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869686f8889749978e8fb1bca3e3b729, da0a02cfb2b04eb685bd7503ff955c25; no voters:
I20260619 12:38:07.092504 14840 raft_consensus.cc:2454] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869686f8889749978e8fb1bca3e3b729 in term 0.
I20260619 12:38:07.092746 15174 raft_consensus.cc:2781] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:38:07.092864 15174 raft_consensus.cc:482] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:38:07.092919 15174 raft_consensus.cc:3037] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:07.093981 15174 raft_consensus.cc:504] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:07.094381 15174 leader_election.cc:288] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [CANDIDATE]: Term 1 election: Requested vote from peers da0a02cfb2b04eb685bd7503ff955c25 (127.0.98.2:37989), fe8ea6b60ebb44268af0f3c9ff0d90ac (127.0.98.1:39831)
I20260619 12:38:07.094767 14978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca7fa19bbee4c8e9cab71ac43898acf" candidate_uuid: "869686f8889749978e8fb1bca3e3b729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da0a02cfb2b04eb685bd7503ff955c25"
I20260619 12:38:07.094769 14840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca7fa19bbee4c8e9cab71ac43898acf" candidate_uuid: "869686f8889749978e8fb1bca3e3b729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac"
I20260619 12:38:07.094902 14978 raft_consensus.cc:3037] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:07.094921 14840 raft_consensus.cc:3037] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:07.096110 14978 raft_consensus.cc:2454] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869686f8889749978e8fb1bca3e3b729 in term 1.
I20260619 12:38:07.096098 14840 raft_consensus.cc:2454] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869686f8889749978e8fb1bca3e3b729 in term 1.
I20260619 12:38:07.096458 15044 leader_election.cc:302] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869686f8889749978e8fb1bca3e3b729, fe8ea6b60ebb44268af0f3c9ff0d90ac; no voters:
I20260619 12:38:07.096625 15174 raft_consensus.cc:2781] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:38:07.096891 15174 raft_consensus.cc:686] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [term 1 LEADER]: Becoming Leader. State: Replica: 869686f8889749978e8fb1bca3e3b729, State: Running, Role: LEADER
I20260619 12:38:07.097246 15174 consensus_queue.cc:237] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:07.100443 14695 catalog_manager.cc:5495] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869686f8889749978e8fb1bca3e3b729 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "869686f8889749978e8fb1bca3e3b729" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } health_report { overall_health: UNKNOWN } } }
I20260619 12:38:07.115935 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.112211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.0.98.3:42635)
0619 12:38:07.112380 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:07.112383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:07.112398 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:38:07.112491 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:07.112493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:07.112549 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:07.112641 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:38:07.112845 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.113352 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.114345 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:07.114569 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:07.114658 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:07.114939 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:38:07.115440 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:38:07.115671 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:07.115754 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 12:38:07.115816 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 12:38:07.122527 14840 raft_consensus.cc:1261] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Refusing update from remote peer 869686f8889749978e8fb1bca3e3b729: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:38:07.122527 14978 raft_consensus.cc:1261] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [term 1 FOLLOWER]: Refusing update from remote peer 869686f8889749978e8fb1bca3e3b729: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260619 12:38:07.122891 14887 tablet.cc:2369] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:38:07.123417 15174 consensus_queue.cc:1035] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:38:07.123711 15176 consensus_queue.cc:1035] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:38:07.131022 15180 mvcc.cc:205] Tried to move back new op lower bound from 7298550526445531136 to 7298550526353100800. Current Snapshot: MvccSnapshot[applied={T|T < 7298550526445531136}]
I20260619 12:38:07.132891 15181 mvcc.cc:205] Tried to move back new op lower bound from 7298550526445531136 to 7298550526353100800. Current Snapshot: MvccSnapshot[applied={T|T < 7298550526445531136}]
I20260619 12:38:07.140519 14697 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:38:07.140753 14697 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:38:07.143710 14697 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc: Sending DeleteTablet for 3 replicas of tablet 2ca7fa19bbee4c8e9cab71ac43898acf
I20260619 12:38:07.144635 14958 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ca7fa19bbee4c8e9cab71ac43898acf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:38:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42264
I20260619 12:38:07.144629 15090 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ca7fa19bbee4c8e9cab71ac43898acf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:38:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54336
I20260619 12:38:07.145248 15190 tablet_replica.cc:330] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: stopping tablet replica
I20260619 12:38:07.145601 15189 tablet_replica.cc:330] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25: stopping tablet replica
I20260619 12:38:07.145754 14820 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ca7fa19bbee4c8e9cab71ac43898acf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:38:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33208
I20260619 12:38:07.145787 15190 raft_consensus.cc:2229] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:38:07.145961 14895 tablet_replica.cc:330] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac: stopping tablet replica
I20260619 12:38:07.145946 15189 raft_consensus.cc:2229] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:38:07.146080 14895 raft_consensus.cc:2229] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:38:07.146232 14895 raft_consensus.cc:2258] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:38:07.146325 15189 raft_consensus.cc:2258] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:38:07.146564 15190 raft_consensus.cc:2258] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:38:07.147150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 14736
I20260619 12:38:07.147400 14895 ts_tablet_manager.cc:1905] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:38:07.150552 15190 ts_tablet_manager.cc:1905] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:38:07.153299 15190 ts_tablet_manager.cc:1918] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:38:07.153534 15190 log.cc:1196] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/2ca7fa19bbee4c8e9cab71ac43898acf
I20260619 12:38:07.153932 15190 ts_tablet_manager.cc:1939] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: Deleting consensus metadata
W20260619 12:38:07.155082 14683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39831 (error 108)
I20260619 12:38:07.155321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 14891
I20260619 12:38:07.156950 14685 catalog_manager.cc:4841] TS 869686f8889749978e8fb1bca3e3b729 (127.0.98.3:42635): tablet 2ca7fa19bbee4c8e9cab71ac43898acf (table test-table [id=bf8d93cf811045aa93e0df2a8005ed2b]) successfully deleted
I20260619 12:38:07.160516 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.160165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39831 (local address 127.0.0.1:33216)
0619 12:38:07.160305 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:07.160392 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 12:38:07.160838 14683 catalog_manager.cc:4584] TS fe8ea6b60ebb44268af0f3c9ff0d90ac (127.0.98.1:39831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ca7fa19bbee4c8e9cab71ac43898acf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39831: connect: Connection refused (error 111)
I20260619 12:38:07.163286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 15027
I20260619 12:38:07.164281 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.163541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37989 (local address 127.0.0.1:42274)
0619 12:38:07.164145 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:07.164200 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501}
I20260619 12:38:07.170027 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 14665
I20260619 12:38:07.176999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42511
--webserver_interface=127.0.98.62
--webserver_port=39205
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:38:07.303637 15192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:07.304080 15192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:07.304180 15192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:07.310456 15192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:38:07.310554 15192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:07.310590 15192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:07.310626 15192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:38:07.310660 15192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:38:07.316397 15192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:07.317536 15192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:07.318560 15192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:07.327355 15198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:07.327395 15192 server_base.cc:1034] running on GCE node
W20260619 12:38:07.327355 15200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:07.327382 15197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:07.328260 15192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:07.329435 15192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:07.330606 15192 hybrid_clock.cc:648] HybridClock initialized: now 1781872687330591 us; error 36 us; skew 500 ppm
Jun 19 12:38:07 dist-test-slave-kdrb krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:07.333567 15192 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:38:07.334594 15192 webserver.cc:460] Webserver started at http://127.0.98.62:39205/ using document root <none> and password file <none>
I20260619 12:38:07.335242 15192 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:07.335309 15192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:07.339594 15192 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:38:07.341858 15207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:07.343179 15192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:38:07.343323 15192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "69c5c649c01847158f9a7230870bd4bc"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "6f26face1203a3ab3a909f799f4a5ddb"
server_key_iv: "1cfa330974990f42586ef0e6987837ef"
server_key_version: "encryptionkey@0"
I20260619 12:38:07.343962 15192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:07.361575 15192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:07.365170 15192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:07.365392 15192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:07.372949 15192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42511
I20260619 12:38:07.373258 15259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42511 every 8 connection(s)
I20260619 12:38:07.374109 15192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:38:07.378999 15260 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:38:07.380820 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc: Bootstrap starting.
I20260619 12:38:07.384408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 15192
I20260619 12:38:07.384928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39831
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42511
--webserver_port=42059
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:38:07.399608 15260 log.cc:824] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:07.411569 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:38:07.412011 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc: Bootstrap complete.
I20260619 12:38:07.416363 15260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } }
I20260619 12:38:07.416782 15260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c5c649c01847158f9a7230870bd4bc, State: Initialized, Role: FOLLOWER
I20260619 12:38:07.417296 15260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } }
I20260619 12:38:07.417413 15260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:38:07.417506 15260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:38:07.417631 15260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 2 FOLLOWER]: Advancing to term 3
I20260619 12:38:07.419831 15260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } }
I20260619 12:38:07.420277 15260 leader_election.cc:302] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69c5c649c01847158f9a7230870bd4bc; no voters:
I20260619 12:38:07.420699 15260 leader_election.cc:288] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 12:38:07.420797 15264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 3 FOLLOWER]: Leader election won for term 3
I20260619 12:38:07.421011 15264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [term 3 LEADER]: Becoming Leader. State: Replica: 69c5c649c01847158f9a7230870bd4bc, State: Running, Role: LEADER
I20260619 12:38:07.421370 15264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } }
I20260619 12:38:07.422160 15260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:38:07.422765 15265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "69c5c649c01847158f9a7230870bd4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } }
I20260619 12:38:07.422899 15265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [sys.catalog]: This master's current role is: LEADER
I20260619 12:38:07.423822 15270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:38:07.423955 15266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69c5c649c01847158f9a7230870bd4bc. Latest consensus state: current_term: 3 leader_uuid: "69c5c649c01847158f9a7230870bd4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c5c649c01847158f9a7230870bd4bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42511 } } }
I20260619 12:38:07.424084 15266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c5c649c01847158f9a7230870bd4bc [sys.catalog]: This master's current role is: LEADER
I20260619 12:38:07.429467 15270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:38:07.430291 15270 catalog_manager.cc:1259] Loaded cluster ID: 1978d65ee6fe40d0bf0abeaea1ef3c82
I20260619 12:38:07.430363 15270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 12:38:07.431356 15270 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe80d0b79ad google::LogMessage::Fail() at ??:0
@ 0x7fe80d0ba88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe80d0b74de google::LogMessage::Flush() at ??:0
@ 0x7fe80d0b8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe8179fcfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe8179fdd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe8179fa579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe817a3e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe80f3093d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe80f309c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe80f30b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe80f2fa512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe8109426db start_thread at ??:0
@ 0x7fe80bbb561f clone at ??:0
W20260619 12:38:07.521733 15262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:07.522078 15262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:07.522174 15262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:07.526759 15262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:07.526852 15262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:07.526942 15262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:38:07.532593 15262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42059
--tserver_master_addrs=127.0.98.62:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:07.533746 15262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:07.534832 15262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:07.545030 15302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:07.545029 15301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:07.545243 15262 server_base.cc:1034] running on GCE node
W20260619 12:38:07.545022 15304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:07.545920 15262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:07.546648 15262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:07.547842 15262 hybrid_clock.cc:648] HybridClock initialized: now 1781872687547813 us; error 48 us; skew 500 ppm
Jun 19 12:38:07 dist-test-slave-kdrb krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:07.550781 15262 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:38:07.551863 15262 webserver.cc:460] Webserver started at http://127.0.98.1:42059/ using document root <none> and password file <none>
I20260619 12:38:07.552505 15262 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:07.552572 15262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:07.556916 15262 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:38:07.559319 15311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:07.560766 15262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:38:07.560925 15262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "5528c98868d005bec62fb8bd75f80dac"
server_key_iv: "d19762f5e07ab6c00848edf20c0a4a8d"
server_key_version: "encryptionkey@0"
I20260619 12:38:07.561584 15262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:07.575824 15262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:07.579404 15262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:07.579634 15262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:07.580317 15262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:38:07.581419 15318 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 12:38:07.587167 15262 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 12:38:07.587280 15262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 12:38:07.587373 15262 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 12:38:07.589133 15262 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 12:38:07.589212 15262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 12:38:07.589536 15318 tablet_bootstrap.cc:492] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac: Bootstrap starting.
I20260619 12:38:07.596716 15318 log.cc:824] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:07.599978 15262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39831
I20260619 12:38:07.600194 15426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39831 every 8 connection(s)
I20260619 12:38:07.600905 15262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:38:07.602109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 15262
I20260619 12:38:07.602512 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37989
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42511
--webserver_port=43471
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:38:07.606786 15318 tablet_bootstrap.cc:492] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:38:07.607225 15318 tablet_bootstrap.cc:492] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac: Bootstrap complete.
I20260619 12:38:07.607640 15318 ts_tablet_manager.cc:1397] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.004s
I20260619 12:38:07.609954 15318 raft_consensus.cc:348] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:07.610399 15318 raft_consensus.cc:729] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8ea6b60ebb44268af0f3c9ff0d90ac, State: Initialized, Role: FOLLOWER
I20260619 12:38:07.610899 15318 consensus_queue.cc:260] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:07.611904 15318 ts_tablet_manager.cc:1428] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
W20260619 12:38:07.699616 15430 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:38:07.604094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:42523)
0619 12:38:07.604659 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:07.604712 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:07.605724 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:07.698761 (+ 93037us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":161,"threads_started":1}
W20260619 12:38:07.700258 15427 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42511 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42511: Client connection negotiation failed: client connection to 127.0.98.62:42511: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 12:38:07.701057 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.700801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:53115)
0619 12:38:07.700931 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:07.700996 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 12:38:07.701731 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.701512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:49487)
0619 12:38:07.701623 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:07.701679 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 12:38:07.701912 15427 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:38:07.737706 15429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:07.738054 15429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:07.738139 15429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:07.742497 15429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:07.742604 15429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:07.742702 15429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:38:07.748245 15429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43471
--tserver_master_addrs=127.0.98.62:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:07.749429 15429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:07.750574 15429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:07.760885 15439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:07.760847 15437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:07.761091 15429 server_base.cc:1034] running on GCE node
W20260619 12:38:07.761037 15436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:07.761607 15429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:07.762332 15429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:07.763557 15429 hybrid_clock.cc:648] HybridClock initialized: now 1781872687763538 us; error 37 us; skew 500 ppm
Jun 19 12:38:07 dist-test-slave-kdrb krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:07.766700 15429 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:38:07.768085 15429 webserver.cc:460] Webserver started at http://127.0.98.2:43471/ using document root <none> and password file <none>
I20260619 12:38:07.768694 15429 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:07.768798 15429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:07.772974 15429 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:38:07.775703 15446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:07.777063 15429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:38:07.777220 15429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "da0a02cfb2b04eb685bd7503ff955c25"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "b7ad49f910b323a226f629ea7481103c"
server_key_iv: "e7de355d38f42f4737a69c9821de6e03"
server_key_version: "encryptionkey@0"
I20260619 12:38:07.777851 15429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:07.789716 15429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:07.793313 15429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:07.793545 15429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:07.794194 15429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:38:07.795301 15453 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 12:38:07.799803 15429 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 12:38:07.799918 15429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 12:38:07.800038 15429 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 12:38:07.801937 15429 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 12:38:07.802023 15429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 12:38:07.802313 15453 tablet_bootstrap.cc:492] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25: Bootstrap starting.
I20260619 12:38:07.813112 15429 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37989
I20260619 12:38:07.813138 15560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37989 every 8 connection(s)
I20260619 12:38:07.814316 15429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:38:07.818080 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.816624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:57777)
0619 12:38:07.817228 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:07.817344 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":162,"threads_started":1}
W20260619 12:38:07.818706 15561 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42511 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42511: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
I20260619 12:38:07.819399 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.819145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:48283)
0619 12:38:07.819270 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:07.819326 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 12:38:07.820163 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.819925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:36369)
0619 12:38:07.820053 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:07.820110 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 12:38:07.820341 15561 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 12:38:07.820943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 15429
I20260619 12:38:07.821352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42635
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42511
--webserver_port=42989
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:38:07.822921 15453 log.cc:824] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:07.833703 15453 tablet_bootstrap.cc:492] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:38:07.834187 15453 tablet_bootstrap.cc:492] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25: Bootstrap complete.
I20260619 12:38:07.835806 15453 ts_tablet_manager.cc:1397] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.011s
I20260619 12:38:07.839850 15453 raft_consensus.cc:348] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:07.840487 15453 raft_consensus.cc:729] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da0a02cfb2b04eb685bd7503ff955c25, State: Initialized, Role: FOLLOWER
I20260619 12:38:07.841089 15453 consensus_queue.cc:260] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:07.842347 15453 ts_tablet_manager.cc:1428] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260619 12:38:07.842504 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.842175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:50367)
0619 12:38:07.842319 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:07.842388 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 12:38:07.843226 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:07.843022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:40047)
0619 12:38:07.843129 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:07.843173 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 12:38:07.942623 15564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:07.942917 15564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:07.942962 15564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:07.947121 15564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:07.947206 15564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:07.947315 15564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:38:07.952817 15564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42989
--tserver_master_addrs=127.0.98.62:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:07.953953 15564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:07.955075 15564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:07.965209 15574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:07.965251 15571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:07.965301 15572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:07.965473 15564 server_base.cc:1034] running on GCE node
I20260619 12:38:07.966027 15564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:07.966722 15564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:07.967916 15564 hybrid_clock.cc:648] HybridClock initialized: now 1781872687967880 us; error 45 us; skew 500 ppm
Jun 19 12:38:07 dist-test-slave-kdrb krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:07.970868 15564 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:38:07.971920 15564 webserver.cc:460] Webserver started at http://127.0.98.3:42989/ using document root <none> and password file <none>
I20260619 12:38:07.972525 15564 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:07.972620 15564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:07.976637 15564 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 12:38:07.978837 15581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:07.980293 15564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 12:38:07.980433 15564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "869686f8889749978e8fb1bca3e3b729"
format_stamp: "Formatted at 2026-06-19 12:38:05 on dist-test-slave-kdrb"
server_key: "701a7555e02c7e40eeb973f5e023011a"
server_key_iv: "3f5df402b22e936c5e082756ace7db9c"
server_key_version: "encryptionkey@0"
I20260619 12:38:07.981051 15564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:07.999152 15564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:08.002158 15564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:08.002380 15564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:08.003005 15564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:38:08.003947 15564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:38:08.004019 15564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:08.004093 15564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:38:08.004137 15564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:08.013525 15564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42635
I20260619 12:38:08.013552 15694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42635 every 8 connection(s)
I20260619 12:38:08.014626 15564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:38:08.018745 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:08.016922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:57547)
0619 12:38:08.017387 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:08.017500 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
I20260619 12:38:08.018908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 15564
W20260619 12:38:08.019362 15695 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42511 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42511: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
I20260619 12:38:08.020092 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:08.019857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:41817)
0619 12:38:08.019961 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:08.020009 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 12:38:08.020766 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:08.020557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:56773)
0619 12:38:08.020655 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:08.020719 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 12:38:08.020933 15695 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 12:38:08.703061 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:08.702572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:57497)
0619 12:38:08.702879 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:08.702959 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":83,"threads_started":1}
I20260619 12:38:08.844496 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:08.844041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:37179)
0619 12:38:08.844333 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:08.844396 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
I20260619 12:38:08.950814 15701 raft_consensus.cc:482] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:38:08.951047 15701 raft_consensus.cc:504] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:08.952283 15701 leader_election.cc:288] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers da0a02cfb2b04eb685bd7503ff955c25 (127.0.98.2:37989), 869686f8889749978e8fb1bca3e3b729 (127.0.98.3:42635)
Jun 19 12:38:08 dist-test-slave-kdrb krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 19 12:38:08 dist-test-slave-kdrb krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260619 12:38:08.967506 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:08.952604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37989 (local address 127.0.98.1:55857)
0619 12:38:08.952775 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:08.952790 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:08.952882 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:08.954411 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:08.954417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:08.955000 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:08.955598 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:08.955615 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:08.956926 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:08.956930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:08.957941 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:08.957951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:08.958826 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:08.958829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:08.959389 (+ 560us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:08.960139 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:08.960166 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:08.962230 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:08.964663 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:08.964669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:08.964685 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:08.965109 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:08.965517 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:08.965520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:08.965522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:08.965686 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:08.966478 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:08.966481 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:08.966689 (+ 208us) client_negotiation.cc:769] Sending connection context
0619 12:38:08.966892 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 12:38:08.967289 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":221,"spinlock_wait_cycles":55296}
I20260619 12:38:08.967502 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:08.952789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55857 (local address 127.0.98.2:37989)
0619 12:38:08.953240 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:08.953256 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:08.953303 (+ 47us) server_negotiation.cc:407] Connection header received
0619 12:38:08.953609 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:08.953614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:08.954019 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:08.954275 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:08.955801 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:08.956753 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:08.958075 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:08.958697 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:08.958875 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:08.962463 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:08.962469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:08.962484 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:08.962525 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:08.964471 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:08.965239 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:08.965249 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:08.965256 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:08.965408 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:08.965786 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:08.965791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:08.965792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:08.966356 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:08.966496 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:08.967002 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 12:38:08.967280 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":108,"threads_started":1}
I20260619 12:38:08.967525 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:08.952789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49529 (local address 127.0.98.3:42635)
0619 12:38:08.953145 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:08.953180 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:08.953348 (+ 168us) server_negotiation.cc:407] Connection header received
0619 12:38:08.954025 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:08.954031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:08.954422 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:08.954725 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:08.955780 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:08.956761 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:08.958076 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:08.958744 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:08.958894 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:08.962594 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:08.962604 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:08.962623 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:08.962661 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:08.964593 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:08.965252 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:08.965257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:08.965261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:08.965418 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:08.965800 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:08.965804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:08.965806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:08.966302 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:08.966442 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:08.967013 (+ 571us) server_negotiation.cc:299] Negotiation successful
0619 12:38:08.967306 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":106,"threads_started":1}
I20260619 12:38:08.967633 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:08.952604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42635 (local address 127.0.98.1:49529)
0619 12:38:08.953229 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:08.953244 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:08.953365 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:08.954684 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:08.954686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:08.954987 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:08.955599 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:08.955616 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:08.956929 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:08.956932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:08.957942 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:08.957952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:08.959078 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:08.959082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:08.959393 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:08.960140 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:08.960179 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:08.962367 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:08.964723 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:08.964729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:08.964734 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:08.965098 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:08.965529 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:08.965532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:08.965534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:08.965700 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:08.966426 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:08.966432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:08.966689 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 12:38:08.966892 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 12:38:08.967380 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":743,"spinlock_wait_cycles":72576,"thread_start_us":66,"threads_started":1}
I20260619 12:38:08.969365 15515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca7fa19bbee4c8e9cab71ac43898acf" candidate_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "da0a02cfb2b04eb685bd7503ff955c25" is_pre_election: true
I20260619 12:38:08.969743 15515 raft_consensus.cc:2454] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8ea6b60ebb44268af0f3c9ff0d90ac in term 1.
W20260619 12:38:08.970476 15315 leader_election.cc:341] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 869686f8889749978e8fb1bca3e3b729 (127.0.98.3:42635): Not found: Tablet not found: 2ca7fa19bbee4c8e9cab71ac43898acf
I20260619 12:38:08.970659 15312 leader_election.cc:302] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: da0a02cfb2b04eb685bd7503ff955c25, fe8ea6b60ebb44268af0f3c9ff0d90ac; no voters: 869686f8889749978e8fb1bca3e3b729
I20260619 12:38:08.970973 15701 raft_consensus.cc:2781] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 12:38:08.971082 15701 raft_consensus.cc:482] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:38:08.971129 15701 raft_consensus.cc:3037] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:38:08.973416 15701 raft_consensus.cc:504] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:08.973807 15701 leader_election.cc:288] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [CANDIDATE]: Term 2 election: Requested vote from peers da0a02cfb2b04eb685bd7503ff955c25 (127.0.98.2:37989), 869686f8889749978e8fb1bca3e3b729 (127.0.98.3:42635)
I20260619 12:38:08.974185 15515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca7fa19bbee4c8e9cab71ac43898acf" candidate_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "da0a02cfb2b04eb685bd7503ff955c25"
I20260619 12:38:08.974366 15515 raft_consensus.cc:3037] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [term 1 FOLLOWER]: Advancing to term 2
W20260619 12:38:08.974445 15315 leader_election.cc:341] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 869686f8889749978e8fb1bca3e3b729 (127.0.98.3:42635): Not found: Tablet not found: 2ca7fa19bbee4c8e9cab71ac43898acf
I20260619 12:38:08.976696 15515 raft_consensus.cc:2454] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8ea6b60ebb44268af0f3c9ff0d90ac in term 2.
I20260619 12:38:08.977064 15312 leader_election.cc:302] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: da0a02cfb2b04eb685bd7503ff955c25, fe8ea6b60ebb44268af0f3c9ff0d90ac; no voters: 869686f8889749978e8fb1bca3e3b729
I20260619 12:38:08.977314 15701 raft_consensus.cc:2781] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 2 FOLLOWER]: Leader election won for term 2
I20260619 12:38:08.977502 15701 raft_consensus.cc:686] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [term 2 LEADER]: Becoming Leader. State: Replica: fe8ea6b60ebb44268af0f3c9ff0d90ac, State: Running, Role: LEADER
I20260619 12:38:08.977862 15701 consensus_queue.cc:237] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:08.978291 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:08.977997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:60831)
0619 12:38:08.978146 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:08.978197 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 12:38:09.022043 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:09.021558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:34541)
0619 12:38:09.021884 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:09.021947 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
W20260619 12:38:09.385743 15315 consensus_peers.cc:487] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac -> Peer 869686f8889749978e8fb1bca3e3b729 (127.0.98.3:42635): Couldn't send request to peer 869686f8889749978e8fb1bca3e3b729. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ca7fa19bbee4c8e9cab71ac43898acf. This is attempt 1: this message will repeat every 5th retry.
I20260619 12:38:09.465451 15515 raft_consensus.cc:1261] T 2ca7fa19bbee4c8e9cab71ac43898acf P da0a02cfb2b04eb685bd7503ff955c25 [term 2 FOLLOWER]: Refusing update from remote peer fe8ea6b60ebb44268af0f3c9ff0d90ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260619 12:38:09.466410 15701 consensus_queue.cc:1035] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260619 12:38:09.468729 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:09.468293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:49557)
0619 12:38:09.468516 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:09.468590 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":58,"threads_started":1}
I20260619 12:38:09.471233 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:09.470940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:57985)
0619 12:38:09.471069 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:09.471137 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 12:38:09.471732 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:09.471509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:55713)
0619 12:38:09.471610 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:09.471659 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 12:38:09.858090 15713 ts_tablet_manager.cc:927] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: Initiating tablet copy from peer fe8ea6b60ebb44268af0f3c9ff0d90ac (127.0.98.1:39831)
I20260619 12:38:09.858695 15713 tablet_copy_client.cc:316] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:39831
Jun 19 12:38:09 dist-test-slave-kdrb krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:38:09.868950 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:09.859282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39831 (local address 127.0.98.3:35407)
0619 12:38:09.859606 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:09.859625 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:09.859747 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:09.860238 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:09.860244 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:09.860617 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:09.860724 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:09.860730 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:09.861646 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:09.861649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:09.862309 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:09.862317 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:09.863066 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:09.863069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:09.863189 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:09.863779 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:09.863791 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:09.865198 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:09.867124 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:09.867134 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:09.867140 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:09.867382 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:09.867709 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:09.867713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:09.867715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:09.867803 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:09.868286 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:09.868292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:09.868427 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 12:38:09.868575 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 12:38:09.868758 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":93,"threads_started":1}
I20260619 12:38:09.868963 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:09.859421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35407 (local address 127.0.98.1:39831)
0619 12:38:09.859808 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:09.859821 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:09.859844 (+ 23us) server_negotiation.cc:407] Connection header received
0619 12:38:09.859892 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:09.859897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:09.860082 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:09.860214 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:09.860834 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:09.861551 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:09.862445 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:09.862929 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:09.863056 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:09.865352 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:09.865362 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:09.865378 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:09.865407 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:09.866997 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:09.867526 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:09.867532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:09.867534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:09.867612 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:09.867897 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:09.867902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:09.867903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:09.868162 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:09.868314 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:09.868662 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 12:38:09.868818 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":171,"threads_started":1}
I20260619 12:38:09.870213 15400 tablet_copy_service.cc:140] P fe8ea6b60ebb44268af0f3c9ff0d90ac: Received BeginTabletCopySession request for tablet 2ca7fa19bbee4c8e9cab71ac43898acf from peer 869686f8889749978e8fb1bca3e3b729 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35407)
I20260619 12:38:09.870388 15400 tablet_copy_service.cc:161] P fe8ea6b60ebb44268af0f3c9ff0d90ac: Beginning new tablet copy session on tablet 2ca7fa19bbee4c8e9cab71ac43898acf from peer 869686f8889749978e8fb1bca3e3b729 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35407: session id = 869686f8889749978e8fb1bca3e3b729-2ca7fa19bbee4c8e9cab71ac43898acf
I20260619 12:38:09.872006 15400 tablet_copy_source_session.cc:205] T 2ca7fa19bbee4c8e9cab71ac43898acf P fe8ea6b60ebb44268af0f3c9ff0d90ac: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 12:38:09.876286 15713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca7fa19bbee4c8e9cab71ac43898acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:09.883376 15713 tablet_copy_client.cc:791] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: tablet copy: Starting download of 0 data blocks...
I20260619 12:38:09.883635 15713 tablet_copy_client.cc:655] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: tablet copy: Starting download of 1 WAL segments...
I20260619 12:38:09.885672 15713 tablet_copy_client.cc:531] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 12:38:09.887555 15713 tablet_bootstrap.cc:492] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: Bootstrap starting.
I20260619 12:38:09.893121 15713 log.cc:824] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:09.900218 15713 tablet_bootstrap.cc:492] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:38:09.900499 15713 tablet_bootstrap.cc:492] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: Bootstrap complete.
I20260619 12:38:09.900784 15713 ts_tablet_manager.cc:1397] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.004s
I20260619 12:38:09.903040 15713 raft_consensus.cc:348] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:09.903301 15713 raft_consensus.cc:729] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869686f8889749978e8fb1bca3e3b729, State: Initialized, Role: FOLLOWER
I20260619 12:38:09.903759 15713 consensus_queue.cc:260] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0a02cfb2b04eb685bd7503ff955c25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37989 } } peers { permanent_uuid: "869686f8889749978e8fb1bca3e3b729" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42635 } } peers { permanent_uuid: "fe8ea6b60ebb44268af0f3c9ff0d90ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } }
I20260619 12:38:09.904700 15713 ts_tablet_manager.cc:1428] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260619 12:38:09.904971 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:09.904692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:57847)
0619 12:38:09.904832 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:09.904895 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 12:38:09.905543 15400 tablet_copy_service.cc:342] P fe8ea6b60ebb44268af0f3c9ff0d90ac: Request end of tablet copy session 869686f8889749978e8fb1bca3e3b729-2ca7fa19bbee4c8e9cab71ac43898acf received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35407
I20260619 12:38:09.905653 15400 tablet_copy_service.cc:433] P fe8ea6b60ebb44268af0f3c9ff0d90ac: ending tablet copy session 869686f8889749978e8fb1bca3e3b729-2ca7fa19bbee4c8e9cab71ac43898acf on tablet 2ca7fa19bbee4c8e9cab71ac43898acf with peer 869686f8889749978e8fb1bca3e3b729
I20260619 12:38:09.905735 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:09.905433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:48459)
0619 12:38:09.905586 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:09.905659 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 12:38:10.445927 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:10.445523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:43309)
0619 12:38:10.445745 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:10.445811 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":61,"threads_started":1}
I20260619 12:38:10.445973 15649 raft_consensus.cc:1203] T 2ca7fa19bbee4c8e9cab71ac43898acf P 869686f8889749978e8fb1bca3e3b729 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260619 12:38:10.447104 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:10.446769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:56957)
0619 12:38:10.446967 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:10.447027 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260619 12:38:10.472921 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:10.472503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:47257)
0619 12:38:10.472757 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:10.472822 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
I20260619 12:38:11.447134 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:11.446724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:55093)
0619 12:38:11.446975 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:11.447038 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
I20260619 12:38:11.448127 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:11.447785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:45821)
0619 12:38:11.447996 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:11.448049 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":62,"threads_started":1}
I20260619 12:38:11.474301 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:11.473776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:40555)
0619 12:38:11.474140 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:11.474202 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":75,"threads_started":1}
I20260619 12:38:12.448678 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:12.448154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:38117)
0619 12:38:12.448468 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:12.448549 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260619 12:38:12.449290 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:12.448869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:49255)
0619 12:38:12.449148 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:12.449197 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
I20260619 12:38:12.475777 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:12.475221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:56885)
0619 12:38:12.475519 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:12.475612 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":107,"threads_started":1}
I20260619 12:38:13.450076 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:13.449578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:47733)
0619 12:38:13.449850 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:13.449919 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260619 12:38:13.450594 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:13.450184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:37369)
0619 12:38:13.450446 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:13.450512 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":69,"threads_started":1}
I20260619 12:38:13.450840 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:13.450605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:49885)
0619 12:38:13.450724 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:13.450774 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 12:38:13.477278 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:13.476821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:34875)
0619 12:38:13.477112 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:13.477177 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
I20260619 12:38:14.452172 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:14.451605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:60663)
0619 12:38:14.451952 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:14.452051 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":107,"threads_started":1}
I20260619 12:38:14.452183 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:14.451603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:49915)
0619 12:38:14.451907 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:14.451974 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":79,"threads_started":1}
I20260619 12:38:14.478607 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:14.478164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:40109)
0619 12:38:14.478437 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:14.478502 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
I20260619 12:38:15.453652 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:15.453171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:42355)
0619 12:38:15.453479 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:15.453552 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":79,"threads_started":1}
I20260619 12:38:15.453668 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:15.453171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:34711)
0619 12:38:15.453511 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:15.453578 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":117,"threads_started":1}
I20260619 12:38:15.479979 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:15.479446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:45773)
0619 12:38:15.479815 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:15.479880 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":123,"threads_started":1}
I20260619 12:38:16.454969 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:16.454514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:54647)
0619 12:38:16.454812 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:16.454875 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":113,"threads_started":1}
I20260619 12:38:16.455132 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:16.454480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:35777)
0619 12:38:16.454921 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:16.454988 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":110,"threads_started":1}
I20260619 12:38:16.481319 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:16.480851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:36421)
0619 12:38:16.481124 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:16.481211 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
I20260619 12:38:17.456264 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:17.455809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:38327)
0619 12:38:17.456102 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:17.456168 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":96,"threads_started":1}
I20260619 12:38:17.456307 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:17.455833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:57537)
0619 12:38:17.456165 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:17.456219 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":86,"threads_started":1}
I20260619 12:38:17.482795 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:17.482276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:37233)
0619 12:38:17.482617 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:17.482687 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":74,"threads_started":1}
I20260619 12:38:18.457602 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:18.457183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:52159)
0619 12:38:18.457458 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:18.457512 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
I20260619 12:38:18.457608 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:18.457154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:56775)
0619 12:38:18.457445 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:18.457505 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
I20260619 12:38:18.484405 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:18.483906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:51195)
0619 12:38:18.484183 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:18.484305 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
I20260619 12:38:19.458979 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:19.458505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:54509)
0619 12:38:19.458792 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:19.458861 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":73,"threads_started":1}
I20260619 12:38:19.458979 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:19.458418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:57777)
0619 12:38:19.458826 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:19.458880 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":87,"threads_started":1}
I20260619 12:38:19.485670 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:19.485225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:50073)
0619 12:38:19.485510 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:19.485571 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":100,"threads_started":1}
I20260619 12:38:20.460414 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:20.459930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:36633)
0619 12:38:20.460246 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:20.460312 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":115,"threads_started":1}
I20260619 12:38:20.460479 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:20.459930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:37665)
0619 12:38:20.460319 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:20.460387 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":116,"threads_started":1}
I20260619 12:38:20.486935 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:20.486504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:53979)
0619 12:38:20.486767 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:20.486834 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260619 12:38:21.461862 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:21.461337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:55713)
0619 12:38:21.461650 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:21.461743 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":117,"threads_started":1}
I20260619 12:38:21.461872 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:21.461337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:52909)
0619 12:38:21.461640 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:21.461745 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
I20260619 12:38:21.488273 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:21.487844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:56897)
0619 12:38:21.488106 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:21.488170 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260619 12:38:22.463331 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:22.462867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.1:58945)
0619 12:38:22.463159 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:22.463231 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":125,"threads_started":1}
I20260619 12:38:22.463344 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:22.462931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.3:59549)
0619 12:38:22.463200 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:22.463256 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":100,"threads_started":1}
I20260619 12:38:22.489708 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:22.489226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42511 (local address 127.0.98.2:53821)
0619 12:38:22.489521 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:22.489579 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":92,"threads_started":1}
I20260619 12:38:23.019804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 15262
I20260619 12:38:23.025876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 15429
I20260619 12:38:23.031410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 15564
2026-06-19T12:38:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19318 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:38:23 dist-test-slave-kdrb krb5kdc[15765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:38:23 dist-test-slave-kdrb krb5kdc[15765](info): set up 2 sockets
Jun 19 12:38:23 dist-test-slave-kdrb krb5kdc[15765](info): commencing operation
krb5kdc: starting...
W20260619 12:38:24.084275 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:38:24 dist-test-slave-kdrb krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:38:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:38:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:38:24.209964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44289
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:38:24.322842 15778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:24.323184 15778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:24.323278 15778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:24.327402 15778 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 12:38:24.327514 15778 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 12:38:24.327546 15778 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 12:38:24.327579 15778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:38:24.327620 15778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:24.327656 15778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:24.327688 15778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:38:24.327723 15778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:38:24.333211 15778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:24.334335 15778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:24.335402 15778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:24.344362 15786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:24.344369 15783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:24.344369 15784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:24.344406 15778 server_base.cc:1034] running on GCE node
I20260619 12:38:24.345316 15778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:24.346498 15778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:24.347653 15778 hybrid_clock.cc:648] HybridClock initialized: now 1781872704347623 us; error 50 us; skew 500 ppm
Jun 19 12:38:24 dist-test-slave-kdrb krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:24.350363 15778 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:38:24.351290 15778 webserver.cc:460] Webserver started at http://127.0.98.62:32995/ using document root <none> and password file <none>
I20260619 12:38:24.351922 15778 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:24.352013 15778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:24.352253 15778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:38:24.354393 15778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "881a73a11f1e42ae88b903eb8cd8d379"
format_stamp: "Formatted at 2026-06-19 12:38:24 on dist-test-slave-kdrb"
server_key: "fe445a6b662e8527536b4e455a700e07"
server_key_iv: "8960d7126318e3441b3933e8297a9e11"
server_key_version: "encryptionkey@0"
I20260619 12:38:24.354965 15778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "881a73a11f1e42ae88b903eb8cd8d379"
format_stamp: "Formatted at 2026-06-19 12:38:24 on dist-test-slave-kdrb"
server_key: "fe445a6b662e8527536b4e455a700e07"
server_key_iv: "8960d7126318e3441b3933e8297a9e11"
server_key_version: "encryptionkey@0"
I20260619 12:38:24.358762 15778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:38:24.361167 15793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:24.362368 15778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:38:24.362505 15778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "881a73a11f1e42ae88b903eb8cd8d379"
format_stamp: "Formatted at 2026-06-19 12:38:24 on dist-test-slave-kdrb"
server_key: "fe445a6b662e8527536b4e455a700e07"
server_key_iv: "8960d7126318e3441b3933e8297a9e11"
server_key_version: "encryptionkey@0"
I20260619 12:38:24.362615 15778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:24.373840 15778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:24.376868 15778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:24.377074 15778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:24.383986 15778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44289
I20260619 12:38:24.384294 15845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44289 every 8 connection(s)
I20260619 12:38:24.384972 15778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:38:24.386595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 15778
I20260619 12:38:24.386698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:38:24.387013 392 external_mini_cluster.cc:1428] Setting key d46e70414c04af0d7941646f705a242d
I20260619 12:38:24.388059 15846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:24.394924 15846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Bootstrap starting.
Jun 19 12:38:24 dist-test-slave-kdrb krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:38:24.397535 15846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:24.398229 15846 log.cc:824] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:24.400323 15846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: No bootstrap required, opened a new log
I20260619 12:38:24.402153 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:24.388451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50654 (local address 127.0.98.62:44289)
0619 12:38:24.388849 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:24.388868 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:24.388910 (+ 42us) server_negotiation.cc:407] Connection header received
0619 12:38:24.389765 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:24.389788 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:24.390193 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:24.390568 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:24.391199 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:24.392755 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:24.393612 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:24.394332 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:24.394487 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:24.397210 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:24.397215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:24.397234 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:24.397263 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:24.399243 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:24.399809 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:24.399821 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:24.399827 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:24.399933 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:24.400229 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:24.400232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:24.400234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:24.400682 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:24.400890 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:24.401151 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 12:38:24.401446 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":105,"threads_started":1}
I20260619 12:38:24.404328 15846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260619 12:38:24.404636 15846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:24.404719 15846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881a73a11f1e42ae88b903eb8cd8d379, State: Initialized, Role: FOLLOWER
I20260619 12:38:24.405129 15846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260619 12:38:24.405289 15846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:38:24.405373 15846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:38:24.405472 15846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:24.406399 15846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260619 12:38:24.406744 15846 leader_election.cc:302] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 881a73a11f1e42ae88b903eb8cd8d379; no voters:
I20260619 12:38:24.407065 15846 leader_election.cc:288] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:38:24.407200 15851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:38:24.407450 15851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 1 LEADER]: Becoming Leader. State: Replica: 881a73a11f1e42ae88b903eb8cd8d379, State: Running, Role: LEADER
I20260619 12:38:24.407842 15851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260619 12:38:24.408368 15846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:38:24.409497 15852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "881a73a11f1e42ae88b903eb8cd8d379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } } }
I20260619 12:38:24.409639 15852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [sys.catalog]: This master's current role is: LEADER
I20260619 12:38:24.409890 15853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 881a73a11f1e42ae88b903eb8cd8d379. Latest consensus state: current_term: 1 leader_uuid: "881a73a11f1e42ae88b903eb8cd8d379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } } }
I20260619 12:38:24.409986 15860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:38:24.410012 15853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [sys.catalog]: This master's current role is: LEADER
I20260619 12:38:24.413298 15860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:38:24.418834 15860 catalog_manager.cc:1347] Generated new cluster ID: 864809e86b524c18a0ecfcd4459678a7
I20260619 12:38:24.418910 15860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:38:24.426833 15860 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:38:24.428076 15860 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:38:24.434461 15860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Generated new TSK 0
I20260619 12:38:24.435452 15860 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:38:24.471217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44289
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:38:24.590334 15872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:24.590626 15872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:24.590670 15872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:24.594842 15872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:24.594926 15872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:24.594988 15872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:38:24.600467 15872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:24.601660 15872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:24.602763 15872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:24.612805 15878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:24.612771 15880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:24.612968 15872 server_base.cc:1034] running on GCE node
W20260619 12:38:24.612818 15877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:24.613551 15872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:24.614214 15872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:24.615372 15872 hybrid_clock.cc:648] HybridClock initialized: now 1781872704615359 us; error 34 us; skew 500 ppm
Jun 19 12:38:24 dist-test-slave-kdrb krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:24.618026 15872 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:38:24.618896 15872 webserver.cc:460] Webserver started at http://127.0.98.1:35077/ using document root <none> and password file <none>
I20260619 12:38:24.619525 15872 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:24.619624 15872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:24.619869 15872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:38:24.622028 15872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "3bcf34e242d1416abf64403baceba773"
format_stamp: "Formatted at 2026-06-19 12:38:24 on dist-test-slave-kdrb"
server_key: "2f0e6c2f87bd59d07e436dc9e733bdac"
server_key_iv: "a3cbdebfaf4392af9e770bfc1820f70f"
server_key_version: "encryptionkey@0"
I20260619 12:38:24.622642 15872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3bcf34e242d1416abf64403baceba773"
format_stamp: "Formatted at 2026-06-19 12:38:24 on dist-test-slave-kdrb"
server_key: "2f0e6c2f87bd59d07e436dc9e733bdac"
server_key_iv: "a3cbdebfaf4392af9e770bfc1820f70f"
server_key_version: "encryptionkey@0"
I20260619 12:38:24.626497 15872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 12:38:24.629210 15887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:24.630517 15872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 12:38:24.630668 15872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "3bcf34e242d1416abf64403baceba773"
format_stamp: "Formatted at 2026-06-19 12:38:24 on dist-test-slave-kdrb"
server_key: "2f0e6c2f87bd59d07e436dc9e733bdac"
server_key_iv: "a3cbdebfaf4392af9e770bfc1820f70f"
server_key_version: "encryptionkey@0"
I20260619 12:38:24.630797 15872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:24.642859 15872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:24.645813 15872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:24.646030 15872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:24.646659 15872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:38:24.647557 15872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:38:24.647621 15872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:24.647692 15872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:38:24.647740 15872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:24.656581 15872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41777
I20260619 12:38:24.656607 16000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41777 every 8 connection(s)
I20260619 12:38:24.657505 15872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 12:38:24 dist-test-slave-kdrb krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:38:24.667776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 15872
I20260619 12:38:24.667920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:38:24.668246 392 external_mini_cluster.cc:1428] Setting key 05244605ad9773fa546947e3cd199786
I20260619 12:38:24.672183 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:24.659360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46199 (local address 127.0.98.62:44289)
0619 12:38:24.659540 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:24.659544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:24.660396 (+ 852us) server_negotiation.cc:407] Connection header received
0619 12:38:24.661926 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:24.661928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:24.661977 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:24.662054 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:24.663210 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:24.663818 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:24.664924 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:24.665145 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:24.665230 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:24.667953 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:24.667972 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:24.667978 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:24.668011 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:24.669750 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:24.670358 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:24.670361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:24.670362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:24.670437 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:24.670839 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:24.670841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:24.670842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:24.671010 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:24.671083 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:24.671802 (+ 719us) server_negotiation.cc:299] Negotiation successful
0619 12:38:24.671961 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:38:24.672950 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:24.659787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:46199)
0619 12:38:24.660220 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:24.660274 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:24.661716 (+ 1442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:24.662232 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:24.662239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:24.662598 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:24.663073 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:24.663086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:24.663931 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:24.663934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:24.664821 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:24.664830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:24.665290 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:24.665293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:24.665432 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:24.666198 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:24.666215 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:24.667765 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:24.669870 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:24.669876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:24.669894 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:24.670262 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:24.670527 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:24.670533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:24.670538 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:24.670746 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:24.671110 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:24.671119 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:24.671524 (+ 405us) client_negotiation.cc:769] Sending connection context
0619 12:38:24.671728 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 12:38:24.672002 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":99,"threads_started":1}
I20260619 12:38:24.674279 16001 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44289
I20260619 12:38:24.674528 16001 heartbeater.cc:464] Registering TS with master...
I20260619 12:38:24.675038 16001 heartbeater.cc:511] Master 127.0.98.62:44289 requested a full tablet report, sending...
I20260619 12:38:24.676596 15810 ts_manager.cc:194] Registered new tserver with Master: 3bcf34e242d1416abf64403baceba773 (127.0.98.1:41777)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:38:24.677843 15810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:38:24.696415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44289
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:38:24.810179 16006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:24.810503 16006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:24.810581 16006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:24.814769 16006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:24.814841 16006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:24.814937 16006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:38:24.820468 16006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:24.821609 16006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:24.822656 16006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:24.832479 16012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:24.832559 16014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:24.832551 16006 server_base.cc:1034] running on GCE node
W20260619 12:38:24.832482 16011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:24.833099 16006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:24.833777 16006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:24.834980 16006 hybrid_clock.cc:648] HybridClock initialized: now 1781872704834945 us; error 47 us; skew 500 ppm
Jun 19 12:38:24 dist-test-slave-kdrb krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:24.837932 16006 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:38:24.838817 16006 webserver.cc:460] Webserver started at http://127.0.98.2:42013/ using document root <none> and password file <none>
I20260619 12:38:24.839350 16006 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:24.839409 16006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:24.839648 16006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:38:24.841675 16006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "cb223adc7c8344a89e4a08ac34131a8f"
format_stamp: "Formatted at 2026-06-19 12:38:24 on dist-test-slave-kdrb"
server_key: "9a250abf6166f45802ff7494254eebbe"
server_key_iv: "43307f89dd55ed5cfb37b82ac87395e9"
server_key_version: "encryptionkey@0"
I20260619 12:38:24.842298 16006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cb223adc7c8344a89e4a08ac34131a8f"
format_stamp: "Formatted at 2026-06-19 12:38:24 on dist-test-slave-kdrb"
server_key: "9a250abf6166f45802ff7494254eebbe"
server_key_iv: "43307f89dd55ed5cfb37b82ac87395e9"
server_key_version: "encryptionkey@0"
I20260619 12:38:24.846478 16006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:38:24.849457 16021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:24.850845 16006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 12:38:24.850951 16006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "cb223adc7c8344a89e4a08ac34131a8f"
format_stamp: "Formatted at 2026-06-19 12:38:24 on dist-test-slave-kdrb"
server_key: "9a250abf6166f45802ff7494254eebbe"
server_key_iv: "43307f89dd55ed5cfb37b82ac87395e9"
server_key_version: "encryptionkey@0"
I20260619 12:38:24.851058 16006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:24.870227 16006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:24.873409 16006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:24.873569 16006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:24.874104 16006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:38:24.874873 16006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:38:24.874919 16006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:24.874960 16006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:38:24.874982 16006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:24.883690 16006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38029
I20260619 12:38:24.883708 16134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38029 every 8 connection(s)
I20260619 12:38:24.884588 16006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:38:24.885357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 16006
I20260619 12:38:24.885483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:38:24.885790 392 external_mini_cluster.cc:1428] Setting key b00f20954b4cde7228d55ebe0f64c194
Jun 19 12:38:24 dist-test-slave-kdrb krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:38:24.899642 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:24.886512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52135 (local address 127.0.98.62:44289)
0619 12:38:24.886655 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:24.886659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:24.887932 (+ 1273us) server_negotiation.cc:407] Connection header received
0619 12:38:24.888837 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:24.888840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:24.888897 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:24.888976 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:24.890200 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:24.890747 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:24.892063 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:24.892308 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:24.892424 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:24.895318 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:24.895322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:24.895324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:24.895352 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:24.896889 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:24.897531 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:24.897536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:24.897538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:24.897637 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:24.898076 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:24.898083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:24.898088 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:24.898385 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:24.898479 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:24.899249 (+ 770us) server_negotiation.cc:299] Negotiation successful
0619 12:38:24.899393 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:38:24.900815 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:24.886965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:52135)
0619 12:38:24.887765 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:24.887819 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:24.888623 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:24.889156 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:24.889163 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:24.889539 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:24.890039 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:24.890054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:24.890883 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:24.890887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:24.891932 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:24.891942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:24.892471 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:24.892475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:24.892652 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:24.893298 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:24.893315 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:24.895117 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:24.897013 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:24.897018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:24.897032 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:24.897404 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:24.897749 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:24.897751 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:24.897753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:24.897937 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:24.898501 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:24.898510 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:24.898913 (+ 403us) client_negotiation.cc:769] Sending connection context
0619 12:38:24.899149 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 12:38:24.899582 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":158,"threads_started":1}
I20260619 12:38:24.902236 16135 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44289
I20260619 12:38:24.902499 16135 heartbeater.cc:464] Registering TS with master...
I20260619 12:38:24.903067 16135 heartbeater.cc:511] Master 127.0.98.62:44289 requested a full tablet report, sending...
I20260619 12:38:24.904377 15810 ts_manager.cc:194] Registered new tserver with Master: cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029)
I20260619 12:38:24.905210 15810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:38:24.914898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44289
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:38:25.030288 16140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:25.030575 16140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:25.030653 16140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:25.034922 16140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:25.034998 16140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:25.035121 16140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:38:25.040733 16140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:25.041914 16140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:25.042985 16140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:25.052942 16148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:25.052913 16145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:25.052904 16146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:25.053166 16140 server_base.cc:1034] running on GCE node
I20260619 12:38:25.053668 16140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:25.054404 16140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:25.055547 16140 hybrid_clock.cc:648] HybridClock initialized: now 1781872705055505 us; error 56 us; skew 500 ppm
Jun 19 12:38:25 dist-test-slave-kdrb krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:25.058404 16140 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:38:25.059316 16140 webserver.cc:460] Webserver started at http://127.0.98.3:35557/ using document root <none> and password file <none>
I20260619 12:38:25.059955 16140 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:25.060065 16140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:25.060297 16140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:38:25.062426 16140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "1c24392cb1a34dd48b840210f3b39198"
format_stamp: "Formatted at 2026-06-19 12:38:25 on dist-test-slave-kdrb"
server_key: "09bd69c7b9a2862cab33f7e4228028db"
server_key_iv: "ec7d9ba69084d0cebbc4b2a6b04af9c6"
server_key_version: "encryptionkey@0"
I20260619 12:38:25.063014 16140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1c24392cb1a34dd48b840210f3b39198"
format_stamp: "Formatted at 2026-06-19 12:38:25 on dist-test-slave-kdrb"
server_key: "09bd69c7b9a2862cab33f7e4228028db"
server_key_iv: "ec7d9ba69084d0cebbc4b2a6b04af9c6"
server_key_version: "encryptionkey@0"
I20260619 12:38:25.067106 16140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:38:25.069754 16155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:25.071050 16140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:38:25.071213 16140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "1c24392cb1a34dd48b840210f3b39198"
format_stamp: "Formatted at 2026-06-19 12:38:25 on dist-test-slave-kdrb"
server_key: "09bd69c7b9a2862cab33f7e4228028db"
server_key_iv: "ec7d9ba69084d0cebbc4b2a6b04af9c6"
server_key_version: "encryptionkey@0"
I20260619 12:38:25.071341 16140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:25.084815 16140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:25.088554 16140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:25.088785 16140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:25.089406 16140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:38:25.090322 16140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:38:25.090394 16140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:25.090463 16140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:38:25.090513 16140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:25.099697 16140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38353
I20260619 12:38:25.099716 16268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38353 every 8 connection(s)
I20260619 12:38:25.100625 16140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:38:25.101285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 16140
I20260619 12:38:25.101439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:38:25.101743 392 external_mini_cluster.cc:1428] Setting key 239743ed9388ac068119ddce08aa02f1
Jun 19 12:38:25 dist-test-slave-kdrb krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:38:25.114593 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:25.102533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36607 (local address 127.0.98.62:44289)
0619 12:38:25.102687 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:25.102692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:25.103701 (+ 1009us) server_negotiation.cc:407] Connection header received
0619 12:38:25.105041 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:25.105045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:25.105097 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:25.105192 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:25.106360 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.106898 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.108229 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.108461 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.108578 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:25.111206 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:25.111210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:25.111212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:25.111239 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:25.112566 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:25.113073 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:25.113077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:25.113078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:25.113166 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:25.113503 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:25.113507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:25.113509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:25.113695 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:25.113804 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:25.114298 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 12:38:25.114422 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:38:25.115718 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:25.103056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:36607)
0619 12:38:25.103550 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:25.103604 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:25.104831 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:25.105364 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:25.105372 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:25.105741 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:25.106219 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:25.106233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.107032 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.107040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:25.108077 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:25.108086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.108618 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.108622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:25.108809 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:25.109511 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:25.109527 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:25.111020 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:25.112688 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:25.112694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:25.112711 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:25.112965 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:25.113265 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:25.113268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:25.113270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:25.113408 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:25.113799 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:25.113805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:25.114063 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 12:38:25.114260 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 12:38:25.114590 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":133,"threads_started":1}
I20260619 12:38:25.116838 16269 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44289
I20260619 12:38:25.117097 16269 heartbeater.cc:464] Registering TS with master...
I20260619 12:38:25.117595 16269 heartbeater.cc:511] Master 127.0.98.62:44289 requested a full tablet report, sending...
I20260619 12:38:25.118659 15810 ts_manager.cc:194] Registered new tserver with Master: 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353)
I20260619 12:38:25.119244 15810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36607
I20260619 12:38:25.127848 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:38:25.137910 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:25.129943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50666 (local address 127.0.98.62:44289)
0619 12:38:25.130098 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:25.130102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:25.130270 (+ 168us) server_negotiation.cc:407] Connection header received
0619 12:38:25.130446 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:25.130449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:25.130502 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:25.130589 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:25.131318 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.131966 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.132928 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.133150 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.133264 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:25.134652 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:25.134676 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:25.134678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:25.134703 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:25.136378 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:25.136751 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:25.136753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:25.136755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:25.136827 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:25.137044 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:25.137046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:25.137047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:25.137264 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:25.137368 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:25.137633 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 12:38:25.137725 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:38:25.140995 15810 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:38:25.144563 15810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:38:25.160279 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:25.154132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.0.98.3:38353)
0619 12:38:25.154963 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:25.154976 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:25.155001 (+ 25us) server_negotiation.cc:407] Connection header received
0619 12:38:25.155081 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:25.155087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:25.155650 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:25.155859 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:25.156156 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.157439 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.158946 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.159579 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.159699 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:25.159855 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:25.160009 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 12:38:25.160107 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":147,"threads_started":1}
I20260619 12:38:25.160270 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:25.153395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38353 (local address 127.0.0.1:44528)
0619 12:38:25.154010 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:25.154068 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:25.154218 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:25.155844 (+ 1626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:25.155847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:25.155866 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:25.155987 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:25.155995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.157618 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.157621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:25.158820 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:25.158827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.159709 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.159712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:25.159820 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:25.159857 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 12:38:25.159978 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 12:38:25.160124 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":40,"thread_start_us":92,"threads_started":1}
I20260619 12:38:25.160574 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:25.153529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38029 (local address 127.0.0.1:35900)
0619 12:38:25.154552 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:25.154568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:25.154657 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:25.155620 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:25.155627 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:25.155649 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:25.155825 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:25.155833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.157209 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.157213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:25.158659 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:25.158669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.159938 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.159942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:25.160105 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:25.160183 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 12:38:25.160265 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 12:38:25.160360 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":53,"thread_start_us":97,"threads_started":1}
I20260619 12:38:25.160817 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:25.154159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41777 (local address 127.0.0.1:34958)
0619 12:38:25.154770 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:25.154785 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:25.154868 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:25.156059 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:25.156062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:25.156078 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:25.156151 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:25.156156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.157616 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.157621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:25.158578 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:25.158596 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.159938 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.159942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:25.160154 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:25.160199 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 12:38:25.160265 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 12:38:25.160547 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":330,"thread_start_us":107,"threads_started":1}
I20260619 12:38:25.161059 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:25.154646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35900 (local address 127.0.98.2:38029)
0619 12:38:25.155024 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:25.155044 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:25.155065 (+ 21us) server_negotiation.cc:407] Connection header received
0619 12:38:25.155180 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:25.155186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:25.155420 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:25.155634 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:25.155969 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.157055 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.158770 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.159607 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.159724 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:25.159867 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:25.160775 (+ 908us) server_negotiation.cc:299] Negotiation successful
0619 12:38:25.160898 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":166,"threads_started":1}
I20260619 12:38:25.161351 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:25.155074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34958 (local address 127.0.98.1:41777)
0619 12:38:25.155366 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:25.155387 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:25.155411 (+ 24us) server_negotiation.cc:407] Connection header received
0619 12:38:25.155530 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:25.155538 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:25.155734 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:25.155889 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:25.156280 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.157430 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.158705 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.159607 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.159724 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:25.159867 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:25.160614 (+ 747us) server_negotiation.cc:299] Negotiation successful
0619 12:38:25.161191 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":108,"threads_started":1}
I20260619 12:38:25.163892 16203 tablet_service.cc:1467] Processing CreateTablet for tablet a4c90e4261174dc088bcd12a16fb2919 (DEFAULT_TABLE table=test-table [id=9af18882918e4fb2bc47d0e6e615019f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:38:25.163985 15935 tablet_service.cc:1467] Processing CreateTablet for tablet a4c90e4261174dc088bcd12a16fb2919 (DEFAULT_TABLE table=test-table [id=9af18882918e4fb2bc47d0e6e615019f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:38:25.165079 16203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c90e4261174dc088bcd12a16fb2919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:25.165078 15935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c90e4261174dc088bcd12a16fb2919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:25.165928 16069 tablet_service.cc:1467] Processing CreateTablet for tablet a4c90e4261174dc088bcd12a16fb2919 (DEFAULT_TABLE table=test-table [id=9af18882918e4fb2bc47d0e6e615019f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:38:25.166914 16069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c90e4261174dc088bcd12a16fb2919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:25.173559 16286 tablet_bootstrap.cc:492] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773: Bootstrap starting.
I20260619 12:38:25.174036 16288 tablet_bootstrap.cc:492] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f: Bootstrap starting.
I20260619 12:38:25.175770 16286 tablet_bootstrap.cc:654] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:25.175810 16287 tablet_bootstrap.cc:492] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198: Bootstrap starting.
I20260619 12:38:25.176403 16288 tablet_bootstrap.cc:654] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:25.176656 16286 log.cc:824] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:25.177294 16288 log.cc:824] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:25.178154 16287 tablet_bootstrap.cc:654] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:25.179117 16286 tablet_bootstrap.cc:492] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773: No bootstrap required, opened a new log
I20260619 12:38:25.179332 16286 ts_tablet_manager.cc:1397] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:38:25.179342 16287 log.cc:824] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:25.179548 16288 tablet_bootstrap.cc:492] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f: No bootstrap required, opened a new log
I20260619 12:38:25.179749 16288 ts_tablet_manager.cc:1397] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:38:25.181007 16287 tablet_bootstrap.cc:492] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198: No bootstrap required, opened a new log
I20260619 12:38:25.181219 16287 ts_tablet_manager.cc:1397] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:38:25.183871 16288 raft_consensus.cc:348] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:25.183871 16286 raft_consensus.cc:348] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:25.184183 16286 raft_consensus.cc:374] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:25.184183 16288 raft_consensus.cc:374] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:25.184248 16288 raft_consensus.cc:729] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb223adc7c8344a89e4a08ac34131a8f, State: Initialized, Role: FOLLOWER
I20260619 12:38:25.184248 16286 raft_consensus.cc:729] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bcf34e242d1416abf64403baceba773, State: Initialized, Role: FOLLOWER
I20260619 12:38:25.184096 16287 raft_consensus.cc:348] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:25.184386 16287 raft_consensus.cc:374] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:25.184461 16287 raft_consensus.cc:729] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c24392cb1a34dd48b840210f3b39198, State: Initialized, Role: FOLLOWER
I20260619 12:38:25.184733 16288 consensus_queue.cc:260] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:25.184733 16286 consensus_queue.cc:260] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:25.184896 16287 consensus_queue.cc:260] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:25.185811 16287 ts_tablet_manager.cc:1428] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 12:38:25.185814 16288 ts_tablet_manager.cc:1428] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:38:25.185814 16286 ts_tablet_manager.cc:1428] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:38:25.186173 16135 heartbeater.cc:503] Master 127.0.98.62:44289 was elected leader, sending a full tablet report...
I20260619 12:38:25.186174 16269 heartbeater.cc:503] Master 127.0.98.62:44289 was elected leader, sending a full tablet report...
I20260619 12:38:25.187247 16001 heartbeater.cc:503] Master 127.0.98.62:44289 was elected leader, sending a full tablet report...
I20260619 12:38:25.188951 16293 raft_consensus.cc:482] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:38:25.189114 16293 raft_consensus.cc:504] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:25.190979 16293 leader_election.cc:288] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353), cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029)
I20260619 12:38:25.195925 16296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:25.191160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38353 (local address 127.0.98.1:47939)
0619 12:38:25.191536 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:25.191553 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:25.191658 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:25.192178 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:25.192181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:25.192209 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:25.192361 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:25.192368 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.193181 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.193184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:25.194828 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:25.194839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.195569 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.195572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:25.195644 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:25.195665 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 12:38:25.195729 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 12:38:25.195790 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":165,"thread_start_us":180,"threads_started":1}
I20260619 12:38:25.195935 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:25.191780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47939 (local address 127.0.98.3:38353)
0619 12:38:25.191887 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:25.191890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:25.191900 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:38:25.191936 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:25.191965 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:25.192012 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:25.192081 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:25.192487 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.193048 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.194981 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.195396 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.195562 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:25.195631 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:25.195743 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 12:38:25.195813 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 12:38:25.196702 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:25.191297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60821 (local address 127.0.98.2:38029)
0619 12:38:25.191662 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:25.191669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:25.191687 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:38:25.191828 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:25.191833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:25.191891 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:25.191973 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:25.192298 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.193052 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.195593 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.196205 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.196301 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:25.196347 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:25.196523 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 12:38:25.196600 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
I20260619 12:38:25.196740 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:25.191223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38029 (local address 127.0.98.1:60821)
0619 12:38:25.191631 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:25.191642 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:25.191712 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:25.191998 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:25.192003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:25.192020 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:25.192170 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:25.192177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.193170 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.193174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:25.194828 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:25.194839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.196310 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.196313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:25.196386 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:25.196405 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 12:38:25.196492 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 12:38:25.196568 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":166,"thread_start_us":113,"threads_started":1}
I20260619 12:38:25.196843 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c90e4261174dc088bcd12a16fb2919" candidate_uuid: "3bcf34e242d1416abf64403baceba773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c24392cb1a34dd48b840210f3b39198" is_pre_election: true
I20260619 12:38:25.197194 16223 raft_consensus.cc:2454] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bcf34e242d1416abf64403baceba773 in term 0.
I20260619 12:38:25.197268 16089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c90e4261174dc088bcd12a16fb2919" candidate_uuid: "3bcf34e242d1416abf64403baceba773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb223adc7c8344a89e4a08ac34131a8f" is_pre_election: true
I20260619 12:38:25.197584 16089 raft_consensus.cc:2454] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bcf34e242d1416abf64403baceba773 in term 0.
I20260619 12:38:25.197727 15891 leader_election.cc:302] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c24392cb1a34dd48b840210f3b39198, 3bcf34e242d1416abf64403baceba773; no voters:
I20260619 12:38:25.198025 16293 raft_consensus.cc:2781] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:38:25.198109 16293 raft_consensus.cc:482] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:38:25.198151 16293 raft_consensus.cc:3037] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:25.199177 16293 raft_consensus.cc:504] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:25.199575 16293 leader_election.cc:288] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 1 election: Requested vote from peers 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353), cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029)
I20260619 12:38:25.199918 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c90e4261174dc088bcd12a16fb2919" candidate_uuid: "3bcf34e242d1416abf64403baceba773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c24392cb1a34dd48b840210f3b39198"
I20260619 12:38:25.199996 16089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c90e4261174dc088bcd12a16fb2919" candidate_uuid: "3bcf34e242d1416abf64403baceba773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb223adc7c8344a89e4a08ac34131a8f"
I20260619 12:38:25.200060 16223 raft_consensus.cc:3037] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:25.200104 16089 raft_consensus.cc:3037] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:25.201076 16089 raft_consensus.cc:2454] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bcf34e242d1416abf64403baceba773 in term 1.
I20260619 12:38:25.201179 16223 raft_consensus.cc:2454] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bcf34e242d1416abf64403baceba773 in term 1.
I20260619 12:38:25.201421 15889 leader_election.cc:302] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bcf34e242d1416abf64403baceba773, cb223adc7c8344a89e4a08ac34131a8f; no voters:
I20260619 12:38:25.201654 16293 raft_consensus.cc:2781] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:38:25.201891 16293 raft_consensus.cc:686] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [term 1 LEADER]: Becoming Leader. State: Replica: 3bcf34e242d1416abf64403baceba773, State: Running, Role: LEADER
I20260619 12:38:25.202236 16293 consensus_queue.cc:237] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:25.205449 15808 catalog_manager.cc:5495] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bcf34e242d1416abf64403baceba773 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3bcf34e242d1416abf64403baceba773" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } health_report { overall_health: HEALTHY } } }
I20260619 12:38:25.226577 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:25.222956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34960 (local address 127.0.98.1:41777)
0619 12:38:25.223106 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:25.223111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:25.223127 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:38:25.223256 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:25.223259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:25.223316 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:25.223431 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:38:25.223722 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.224230 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.225099 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:25.225326 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:25.225437 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:25.225753 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:38:25.226191 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:38:25.226333 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:25.226368 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 12:38:25.226425 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 12:38:25.232504 16223 raft_consensus.cc:1261] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198 [term 1 FOLLOWER]: Refusing update from remote peer 3bcf34e242d1416abf64403baceba773: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:38:25.232506 16089 raft_consensus.cc:1261] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f [term 1 FOLLOWER]: Refusing update from remote peer 3bcf34e242d1416abf64403baceba773: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:38:25.233230 16297 consensus_queue.cc:1035] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:38:25.233430 16293 consensus_queue.cc:1035] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:38:25.249380 15810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:38:25.249533 15810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:38:25.251901 15810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Sending DeleteTablet for 3 replicas of tablet a4c90e4261174dc088bcd12a16fb2919
I20260619 12:38:25.252664 16203 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c90e4261174dc088bcd12a16fb2919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:38:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44528
I20260619 12:38:25.252815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 15872
I20260619 12:38:25.252831 16069 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c90e4261174dc088bcd12a16fb2919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:38:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35900
I20260619 12:38:25.253803 16311 tablet_replica.cc:330] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f: stopping tablet replica
I20260619 12:38:25.254148 16311 raft_consensus.cc:2229] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:38:25.254488 16311 raft_consensus.cc:2258] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:38:25.254695 16310 tablet_replica.cc:330] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198: stopping tablet replica
I20260619 12:38:25.255010 16310 raft_consensus.cc:2229] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:38:25.255311 16310 raft_consensus.cc:2258] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:38:25.256131 16311 ts_tablet_manager.cc:1905] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:38:25.256886 16310 ts_tablet_manager.cc:1905] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:38:25.258476 16311 ts_tablet_manager.cc:1918] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:38:25.258589 16311 log.cc:1196] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/a4c90e4261174dc088bcd12a16fb2919
I20260619 12:38:25.258885 16311 ts_tablet_manager.cc:1939] T a4c90e4261174dc088bcd12a16fb2919 P cb223adc7c8344a89e4a08ac34131a8f: Deleting consensus metadata
I20260619 12:38:25.260138 15795 catalog_manager.cc:4841] TS cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029): tablet a4c90e4261174dc088bcd12a16fb2919 (table test-table [id=9af18882918e4fb2bc47d0e6e615019f]) successfully deleted
I20260619 12:38:25.260260 16310 ts_tablet_manager.cc:1918] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:38:25.260353 16310 log.cc:1196] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/a4c90e4261174dc088bcd12a16fb2919
W20260619 12:38:25.260404 15795 connection.cc:535] client connection to 127.0.98.1:41777 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 12:38:25.260557 15795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 12:38:25.260591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 16006
I20260619 12:38:25.260726 16310 ts_tablet_manager.cc:1939] T a4c90e4261174dc088bcd12a16fb2919 P 1c24392cb1a34dd48b840210f3b39198: Deleting consensus metadata
I20260619 12:38:25.262442 15797 catalog_manager.cc:4841] TS 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353): tablet a4c90e4261174dc088bcd12a16fb2919 (table test-table [id=9af18882918e4fb2bc47d0e6e615019f]) successfully deleted
I20260619 12:38:25.263324 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:25.262950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41777 (local address 127.0.0.1:34968)
0619 12:38:25.263118 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:25.263211 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 12:38:25.263531 15795 catalog_manager.cc:4584] TS 3bcf34e242d1416abf64403baceba773 (127.0.98.1:41777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4c90e4261174dc088bcd12a16fb2919: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41777: connect: Connection refused (error 111)
I20260619 12:38:25.266978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 16140
I20260619 12:38:25.272316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 15778
I20260619 12:38:45.278697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44289
--webserver_interface=127.0.98.62
--webserver_port=32995
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:38:45.395771 16313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:45.396149 16313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:45.396236 16313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:45.400664 16313 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 12:38:45.400770 16313 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 12:38:45.400799 16313 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 12:38:45.400822 16313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:38:45.400866 16313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:45.400913 16313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:45.400969 16313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:38:45.401010 16313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:38:45.406821 16313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=32995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:45.408227 16313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:45.409374 16313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:38:45.424239 16313 server_base.cc:1034] running on GCE node
W20260619 12:38:45.424214 16319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:45.424449 16320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:45.424213 16322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:45.425252 16313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:45.426610 16313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:45.427812 16313 hybrid_clock.cc:648] HybridClock initialized: now 1781872725427800 us; error 31 us; skew 500 ppm
Jun 19 12:38:45 dist-test-slave-kdrb krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:45.434906 16313 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:38:45.436215 16313 webserver.cc:460] Webserver started at http://127.0.98.62:32995/ using document root <none> and password file <none>
I20260619 12:38:45.436853 16313 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:45.436940 16313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:45.441358 16313 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 12:38:45.444191 16329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:45.445628 16313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:38:45.445786 16313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "881a73a11f1e42ae88b903eb8cd8d379"
format_stamp: "Formatted at 2026-06-19 12:38:24 on dist-test-slave-kdrb"
server_key: "fe445a6b662e8527536b4e455a700e07"
server_key_iv: "8960d7126318e3441b3933e8297a9e11"
server_key_version: "encryptionkey@0"
I20260619 12:38:45.446475 16313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:45.474123 16313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:45.477645 16313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:45.477872 16313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:45.485626 16313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44289
I20260619 12:38:45.485966 16381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44289 every 8 connection(s)
I20260619 12:38:45.486862 16313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:38:45.491706 16382 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:38:45.493635 16382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Bootstrap starting.
I20260619 12:38:45.496274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 16313
I20260619 12:38:45.496814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41777
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44289
--webserver_port=35077
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:38:45.517117 16382 log.cc:824] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:45.528841 16382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:38:45.529323 16382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Bootstrap complete.
I20260619 12:38:45.534977 16382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260619 12:38:45.535683 16382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881a73a11f1e42ae88b903eb8cd8d379, State: Initialized, Role: FOLLOWER
I20260619 12:38:45.536290 16382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260619 12:38:45.536463 16382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:38:45.536528 16382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:38:45.536624 16382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:38:45.539279 16382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260619 12:38:45.539716 16382 leader_election.cc:302] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 881a73a11f1e42ae88b903eb8cd8d379; no voters:
I20260619 12:38:45.540091 16382 leader_election.cc:288] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 12:38:45.540282 16386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 12:38:45.540580 16386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 2 LEADER]: Becoming Leader. State: Replica: 881a73a11f1e42ae88b903eb8cd8d379, State: Running, Role: LEADER
I20260619 12:38:45.540958 16386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260619 12:38:45.541566 16382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:38:45.542673 16387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "881a73a11f1e42ae88b903eb8cd8d379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } } }
I20260619 12:38:45.542685 16388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 881a73a11f1e42ae88b903eb8cd8d379. Latest consensus state: current_term: 2 leader_uuid: "881a73a11f1e42ae88b903eb8cd8d379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } } }
I20260619 12:38:45.542826 16388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [sys.catalog]: This master's current role is: LEADER
I20260619 12:38:45.542826 16387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [sys.catalog]: This master's current role is: LEADER
I20260619 12:38:45.543299 16395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:38:45.548638 16395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:38:45.549458 16395 catalog_manager.cc:1259] Loaded cluster ID: 864809e86b524c18a0ecfcd4459678a7
I20260619 12:38:45.549528 16395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:38:45.552186 16395 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:38:45.552965 16395 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Loaded TSK: 0
I20260619 12:38:45.569151 16395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Generated new TSK 1
I20260619 12:38:45.569470 16395 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:38:45.632758 16384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:45.633103 16384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:45.633183 16384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:45.637713 16384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:45.637822 16384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:45.637919 16384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:38:45.643885 16384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35077
--tserver_master_addrs=127.0.98.62:44289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:45.645030 16384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:45.646137 16384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:45.656368 16409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:45.656390 16410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:45.656780 16384 server_base.cc:1034] running on GCE node
W20260619 12:38:45.656852 16412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:45.657279 16384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:45.658093 16384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:45.659291 16384 hybrid_clock.cc:648] HybridClock initialized: now 1781872725659263 us; error 61 us; skew 500 ppm
Jun 19 12:38:45 dist-test-slave-kdrb krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:45.662542 16384 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:38:45.663650 16384 webserver.cc:460] Webserver started at http://127.0.98.1:35077/ using document root <none> and password file <none>
I20260619 12:38:45.664287 16384 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:45.664407 16384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:45.668913 16384 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 12:38:45.671542 16419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:45.673028 16384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 12:38:45.673199 16384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "3bcf34e242d1416abf64403baceba773"
format_stamp: "Formatted at 2026-06-19 12:38:24 on dist-test-slave-kdrb"
server_key: "2f0e6c2f87bd59d07e436dc9e733bdac"
server_key_iv: "a3cbdebfaf4392af9e770bfc1820f70f"
server_key_version: "encryptionkey@0"
I20260619 12:38:45.673834 16384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:45.687189 16384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:45.690927 16384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:45.691152 16384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:45.691859 16384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:38:45.693012 16426 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 12:38:45.698700 16384 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 12:38:45.698827 16384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 12:38:45.698925 16384 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 12:38:45.700842 16384 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 12:38:45.700929 16384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 12:38:45.701253 16426 tablet_bootstrap.cc:492] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773: Bootstrap starting.
I20260619 12:38:45.711602 16384 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41777
I20260619 12:38:45.711651 16533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41777 every 8 connection(s)
I20260619 12:38:45.712652 16384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:38:45.715372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 16384
I20260619 12:38:45.715907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38029
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44289
--webserver_port=42013
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:38:45.721736 16426 log.cc:824] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:38:45 dist-test-slave-kdrb krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:38:45.732008 16426 tablet_bootstrap.cc:492] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:38:45.732538 16426 tablet_bootstrap.cc:492] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773: Bootstrap complete.
I20260619 12:38:45.732697 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:45.715190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39731 (local address 127.0.98.62:44289)
0619 12:38:45.716687 (+ 1497us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:45.716710 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:45.716784 (+ 74us) server_negotiation.cc:407] Connection header received
0619 12:38:45.717822 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:45.717841 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:45.718307 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:45.718698 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:45.720383 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:45.722170 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:45.723634 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:45.724144 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:45.724339 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:45.726917 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:45.726922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:45.726937 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:45.726968 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:45.728948 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:45.729728 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:45.729734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:45.729740 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:45.729892 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:45.730301 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:45.730303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:45.730305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:45.730664 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:45.730817 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:45.731518 (+ 701us) server_negotiation.cc:299] Negotiation successful
0619 12:38:45.731812 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1316,"thread_start_us":145,"threads_started":1}
I20260619 12:38:45.732717 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:45.715142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:39731)
0619 12:38:45.715964 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:45.716027 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:45.717280 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:45.718820 (+ 1540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:45.718830 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:45.719438 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:45.720171 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:45.720193 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:45.722365 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:45.722372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:45.723433 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:45.723445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:45.724311 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:45.724315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:45.724519 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:45.725173 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:45.725190 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:45.726685 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:45.729084 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:45.729090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:45.729106 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:45.729492 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:45.730007 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:45.730012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:45.730017 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:45.730198 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:45.730787 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:45.730794 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:45.731124 (+ 330us) client_negotiation.cc:769] Sending connection context
0619 12:38:45.731346 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 12:38:45.731727 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":152,"threads_started":1}
I20260619 12:38:45.734050 16426 ts_tablet_manager.cc:1397] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.009s
I20260619 12:38:45.735556 16534 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44289
I20260619 12:38:45.735880 16534 heartbeater.cc:464] Registering TS with master...
I20260619 12:38:45.736428 16534 heartbeater.cc:511] Master 127.0.98.62:44289 requested a full tablet report, sending...
I20260619 12:38:45.737313 16426 raft_consensus.cc:348] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:45.737864 16426 raft_consensus.cc:729] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bcf34e242d1416abf64403baceba773, State: Initialized, Role: FOLLOWER
I20260619 12:38:45.738400 16426 consensus_queue.cc:260] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:45.739095 16346 ts_manager.cc:194] Registered new tserver with Master: 3bcf34e242d1416abf64403baceba773 (127.0.98.1:41777)
I20260619 12:38:45.739498 16426 ts_tablet_manager.cc:1428] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260619 12:38:45.743233 16346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39731
I20260619 12:38:45.746366 16534 heartbeater.cc:503] Master 127.0.98.62:44289 was elected leader, sending a full tablet report...
Jun 19 12:38:45 dist-test-slave-kdrb krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:38:45.753374 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:45.742268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.0.98.1:41777)
0619 12:38:45.743189 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:45.743238 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:45.743266 (+ 28us) server_negotiation.cc:407] Connection header received
0619 12:38:45.743361 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:45.743366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:45.743601 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:45.743745 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:45.744527 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:45.745287 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:45.746217 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:45.746652 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:45.746947 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:45.749497 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:45.749514 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:45.749525 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:45.749550 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:45.751246 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:45.751869 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:45.751874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:45.751875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:45.751955 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:45.752241 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:45.752246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:45.752248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:45.752522 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:45.752671 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:45.753037 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 12:38:45.753222 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":133,"threads_started":1}
I20260619 12:38:45.753374 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:45.742245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41777 (local address 127.0.0.1:46722)
0619 12:38:45.742713 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:45.742775 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:45.742920 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:45.743780 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:45.743786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:45.744268 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:45.744405 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:45.744417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:45.745415 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:45.745420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:45.746064 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:45.746073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:45.746931 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:45.746953 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:45.747078 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:45.747776 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:45.747791 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:45.749348 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:45.751374 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:45.751383 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:45.751389 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:45.751726 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:45.752052 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:45.752055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:45.752058 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:45.752149 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:45.752643 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:45.752649 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:45.752797 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 12:38:45.752941 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 12:38:45.753178 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":135,"threads_started":1}
I20260619 12:38:45.754454 16468 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c90e4261174dc088bcd12a16fb2919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:38:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46722
I20260619 12:38:45.755514 16543 tablet_replica.cc:330] stopping tablet replica
I20260619 12:38:45.755854 16543 raft_consensus.cc:2229] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:38:45.756134 16543 raft_consensus.cc:2258] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:38:45.757164 16543 ts_tablet_manager.cc:1905] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:38:45.760955 16543 ts_tablet_manager.cc:1918] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:38:45.761049 16543 log.cc:1196] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal/wals/a4c90e4261174dc088bcd12a16fb2919
I20260619 12:38:45.761415 16543 ts_tablet_manager.cc:1939] T a4c90e4261174dc088bcd12a16fb2919 P 3bcf34e242d1416abf64403baceba773: Deleting consensus metadata
I20260619 12:38:45.763291 16331 catalog_manager.cc:4841] TS 3bcf34e242d1416abf64403baceba773 (127.0.98.1:41777): tablet a4c90e4261174dc088bcd12a16fb2919 (table test-table [id=9af18882918e4fb2bc47d0e6e615019f]) successfully deleted
W20260619 12:38:45.841326 16538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:45.841652 16538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:45.841748 16538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:45.846177 16538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:45.846275 16538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:45.846371 16538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:38:45.852061 16538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42013
--tserver_master_addrs=127.0.98.62:44289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:45.853199 16538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:45.854249 16538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:38:45.864253 16551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:45.864374 16549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:45.864413 16548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:45.864703 16538 server_base.cc:1034] running on GCE node
I20260619 12:38:45.865124 16538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:45.865777 16538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:45.866961 16538 hybrid_clock.cc:648] HybridClock initialized: now 1781872725866917 us; error 59 us; skew 500 ppm
Jun 19 12:38:45 dist-test-slave-kdrb krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:45.869792 16538 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:38:45.870697 16538 webserver.cc:460] Webserver started at http://127.0.98.2:42013/ using document root <none> and password file <none>
I20260619 12:38:45.871269 16538 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:45.871380 16538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:45.875286 16538 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:38:45.877425 16558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:45.878750 16538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:38:45.878882 16538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "cb223adc7c8344a89e4a08ac34131a8f"
format_stamp: "Formatted at 2026-06-19 12:38:24 on dist-test-slave-kdrb"
server_key: "9a250abf6166f45802ff7494254eebbe"
server_key_iv: "43307f89dd55ed5cfb37b82ac87395e9"
server_key_version: "encryptionkey@0"
I20260619 12:38:45.879431 16538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:45.896394 16538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:45.900048 16538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:45.900269 16538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:45.900897 16538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:38:45.901734 16538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:38:45.901803 16538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:45.901875 16538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:38:45.901923 16538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:45.910930 16538 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38029
I20260619 12:38:45.910971 16671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38029 every 8 connection(s)
I20260619 12:38:45.912031 16538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:38:45.914340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 16538
I20260619 12:38:45.914839 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38353
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44289
--webserver_port=35557
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:38:45 dist-test-slave-kdrb krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:38:45.929145 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:45.913912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37127 (local address 127.0.98.62:44289)
0619 12:38:45.914084 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:45.914087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:45.915160 (+ 1073us) server_negotiation.cc:407] Connection header received
0619 12:38:45.916913 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:45.916916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:45.916972 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:45.917052 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:45.918431 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:45.918990 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:45.921117 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:45.921364 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:45.921461 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:45.924592 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:45.924598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:45.924602 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:45.924640 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:45.926696 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:45.927261 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:45.927266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:45.927270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:45.927374 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:45.927889 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:45.927894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:45.927896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:45.928137 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:45.928233 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:45.928815 (+ 582us) server_negotiation.cc:299] Negotiation successful
0619 12:38:45.928956 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 12:38:45.930125 16674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:45.914406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:37127)
0619 12:38:45.914973 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:45.915044 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:45.916635 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:45.917294 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:45.917307 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:45.917751 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:45.918259 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:45.918273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:45.919178 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:45.919185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:45.920961 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:45.920977 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:45.921555 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:45.921560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:45.921819 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:45.922543 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:45.922572 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:45.924395 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:45.926843 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:45.926848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:45.926862 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:45.927139 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:45.927574 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:45.927576 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:45.927578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:45.927750 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:45.928239 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:45.928244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:45.928532 (+ 288us) client_negotiation.cc:769] Sending connection context
0619 12:38:45.928733 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 12:38:45.929085 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":160,"threads_started":1}
I20260619 12:38:45.931384 16672 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44289
I20260619 12:38:45.931658 16672 heartbeater.cc:464] Registering TS with master...
I20260619 12:38:45.932186 16672 heartbeater.cc:511] Master 127.0.98.62:44289 requested a full tablet report, sending...
I20260619 12:38:45.933290 16346 ts_manager.cc:194] Registered new tserver with Master: cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029)
I20260619 12:38:45.933878 16346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37127
W20260619 12:38:46.034297 16675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:38:46.034595 16675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:38:46.034677 16675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:38:46.039079 16675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:38:46.039163 16675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:38:46.039261 16675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:38:46.045228 16675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35557
--tserver_master_addrs=127.0.98.62:44289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:38:46.046397 16675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:38:46.047502 16675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:38:46.058080 16675 server_base.cc:1034] running on GCE node
W20260619 12:38:46.058024 16680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:46.058008 16681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:38:46.058033 16683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:38:46.058764 16675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:38:46.059445 16675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:38:46.060688 16675 hybrid_clock.cc:648] HybridClock initialized: now 1781872726060659 us; error 40 us; skew 500 ppm
Jun 19 12:38:46 dist-test-slave-kdrb krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:38:46.063563 16675 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:38:46.064486 16675 webserver.cc:460] Webserver started at http://127.0.98.3:35557/ using document root <none> and password file <none>
I20260619 12:38:46.065074 16675 fs_manager.cc:359] Metadata directory not provided
I20260619 12:38:46.065188 16675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:38:46.069249 16675 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:38:46.071514 16690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:46.072863 16675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:38:46.073004 16675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "1c24392cb1a34dd48b840210f3b39198"
format_stamp: "Formatted at 2026-06-19 12:38:25 on dist-test-slave-kdrb"
server_key: "09bd69c7b9a2862cab33f7e4228028db"
server_key_iv: "ec7d9ba69084d0cebbc4b2a6b04af9c6"
server_key_version: "encryptionkey@0"
I20260619 12:38:46.073567 16675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:38:46.094902 16675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:38:46.098433 16675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:38:46.098649 16675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:38:46.099298 16675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:38:46.100162 16675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:38:46.100208 16675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:46.100283 16675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:38:46.100325 16675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:38:46.109526 16675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38353
I20260619 12:38:46.109570 16803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38353 every 8 connection(s)
I20260619 12:38:46.110576 16675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:38:46.113160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 16675
Jun 19 12:38:46 dist-test-slave-kdrb krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:38:46.124162 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:46.114224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46658 (local address 127.0.98.62:44289)
0619 12:38:46.114474 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:46.114477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:46.114489 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:38:46.114528 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:46.114530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:46.114566 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:46.114677 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:46.115582 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.116130 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.118393 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.118673 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.118748 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:46.120340 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:46.120346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:46.120351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:46.120383 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:46.122202 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:46.122759 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:46.122763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:46.122764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:46.122844 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:46.123146 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:46.123149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:46.123150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:46.123342 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:46.123553 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:46.123811 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 12:38:46.123957 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":185,"spinlock_wait_cycles":3840,"thread_start_us":97,"threads_started":1}
I20260619 12:38:46.125417 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:46.112564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38609 (local address 127.0.98.62:44289)
0619 12:38:46.112723 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:46.112726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:46.113649 (+ 923us) server_negotiation.cc:407] Connection header received
0619 12:38:46.115071 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:46.115076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:46.115129 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:46.115222 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:46.116474 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.117215 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.118396 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.118689 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.118816 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:46.121444 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:46.121447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:46.121449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:46.121481 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:46.123098 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:46.123743 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:46.123746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:46.123747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:46.123834 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:46.124227 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:46.124230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:46.124232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:46.124421 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:46.124526 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:46.125134 (+ 608us) server_negotiation.cc:299] Negotiation successful
0619 12:38:46.125262 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":71,"spinlock_wait_cycles":1408}
I20260619 12:38:46.126521 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:46.113010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:38609)
0619 12:38:46.113474 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:46.113528 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:46.114862 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:46.115409 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:46.115416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:46.115831 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:38:46.116340 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:46.116354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.117344 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.117347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:46.118265 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:46.118274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.118898 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.118902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:46.119066 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:46.119771 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:38:46.119787 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:38:46.121266 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:38:46.123235 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:46.123240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:46.123255 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:46.123633 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:46.123940 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:38:46.123943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:38:46.123945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:38:46.124138 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:38:46.124545 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:38:46.124553 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:38:46.124882 (+ 329us) client_negotiation.cc:769] Sending connection context
0619 12:38:46.125082 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 12:38:46.125442 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":123,"threads_started":1}
I20260619 12:38:46.127718 16804 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44289
I20260619 12:38:46.128041 16804 heartbeater.cc:464] Registering TS with master...
I20260619 12:38:46.128690 16804 heartbeater.cc:511] Master 127.0.98.62:44289 requested a full tablet report, sending...
I20260619 12:38:46.129868 16346 ts_manager.cc:194] Registered new tserver with Master: 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353)
I20260619 12:38:46.130443 16346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38609
I20260619 12:38:46.138017 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:38:46.142385 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:46.138776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.0.98.62:44289)
0619 12:38:46.138906 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:46.138909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:46.139138 (+ 229us) server_negotiation.cc:407] Connection header received
0619 12:38:46.139309 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:46.139311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:46.139420 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:46.139595 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:38:46.139817 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.140332 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.141176 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.141385 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.141457 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:46.141718 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:38:46.141924 (+ 206us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 12:38:46.142203 (+ 279us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46674: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 12:38:46.142473 16536 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46674: token expired
W20260619 12:38:46.143826 16274 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:44289)
I20260619 12:38:46.153759 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:46.144349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46682 (local address 127.0.98.62:44289)
0619 12:38:46.144498 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:46.144504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:46.144545 (+ 41us) server_negotiation.cc:407] Connection header received
0619 12:38:46.144728 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:46.144733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:46.144790 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:46.144896 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:38:46.146260 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.147204 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.148152 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.148589 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.148710 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:46.150473 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:38:46.150476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:38:46.150479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:38:46.150505 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:38:46.152171 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:46.152613 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:46.152615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:46.152616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:46.152699 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:38:46.152947 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:38:46.152950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:38:46.152952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:38:46.153146 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:38:46.153248 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:46.153450 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 12:38:46.153557 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:38:46.156464 16346 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:38:46.159960 16346 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:38:46.169064 16468 tablet_service.cc:1467] Processing CreateTablet for tablet d310b00f6c3e4259a2f33e36af4f047c (DEFAULT_TABLE table=test-table [id=b5e5a1010ccb4d189701f9b8da33704b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:38:46.169796 16468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d310b00f6c3e4259a2f33e36af4f047c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:46.172928 16426 tablet_bootstrap.cc:492] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773: Bootstrap starting.
I20260619 12:38:46.173660 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:46.167692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38353 (local address 127.0.0.1:56434)
0619 12:38:46.167806 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:46.167824 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:46.167914 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:46.168907 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:46.168910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:46.168929 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:46.169062 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:46.169073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.170450 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.170455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:46.171785 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:46.171796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.173152 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.173155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:46.173287 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:46.173345 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 12:38:46.173400 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 12:38:46.173496 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":1}
I20260619 12:38:46.173893 16426 tablet_bootstrap.cc:654] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:46.174386 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:46.167893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56434 (local address 127.0.98.3:38353)
0619 12:38:46.168315 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:46.168331 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:46.168373 (+ 42us) server_negotiation.cc:407] Connection header received
0619 12:38:46.168465 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:46.168471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:46.168746 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:46.168935 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:46.169193 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.170337 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.171928 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.173007 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.173972 (+ 965us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:46.174134 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:46.174182 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 12:38:46.174272 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":218,"threads_started":1}
I20260619 12:38:46.174966 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:46.167968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38029 (local address 127.0.0.1:56824)
0619 12:38:46.168666 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:46.168681 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:46.168768 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:46.170476 (+ 1708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:46.170483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:46.170505 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:46.170599 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:46.170614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.171989 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.171995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:46.173166 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:46.173177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.174555 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.174560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:46.174652 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:46.174678 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 12:38:46.174758 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 12:38:46.174839 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":154,"threads_started":1}
I20260619 12:38:46.175583 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:46.168251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.0.98.2:38029)
0619 12:38:46.168544 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:46.169466 (+ 922us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:46.169651 (+ 185us) server_negotiation.cc:407] Connection header received
0619 12:38:46.169734 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:46.169739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:46.169954 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:46.170247 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:46.170746 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.171901 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.173308 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.174196 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.174328 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:46.174481 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:46.175282 (+ 801us) server_negotiation.cc:299] Negotiation successful
0619 12:38:46.175407 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":96,"threads_started":1}
I20260619 12:38:46.175695 16426 tablet_bootstrap.cc:492] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773: No bootstrap required, opened a new log
I20260619 12:38:46.175786 16426 ts_tablet_manager.cc:1397] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:38:46.176223 16426 raft_consensus.cc:348] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:46.176357 16426 raft_consensus.cc:374] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:46.176419 16426 raft_consensus.cc:729] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bcf34e242d1416abf64403baceba773, State: Initialized, Role: FOLLOWER
I20260619 12:38:46.176581 16426 consensus_queue.cc:260] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:46.177323 16426 ts_tablet_manager.cc:1428] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:38:46.177845 16738 tablet_service.cc:1467] Processing CreateTablet for tablet d310b00f6c3e4259a2f33e36af4f047c (DEFAULT_TABLE table=test-table [id=b5e5a1010ccb4d189701f9b8da33704b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:38:46.178609 16606 tablet_service.cc:1467] Processing CreateTablet for tablet d310b00f6c3e4259a2f33e36af4f047c (DEFAULT_TABLE table=test-table [id=b5e5a1010ccb4d189701f9b8da33704b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:38:46.179055 16738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d310b00f6c3e4259a2f33e36af4f047c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:46.179711 16606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d310b00f6c3e4259a2f33e36af4f047c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:38:46.188217 16817 tablet_bootstrap.cc:492] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f: Bootstrap starting.
I20260619 12:38:46.188217 16818 tablet_bootstrap.cc:492] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198: Bootstrap starting.
I20260619 12:38:46.191308 16817 tablet_bootstrap.cc:654] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:46.191308 16818 tablet_bootstrap.cc:654] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198: Neither blocks nor log segments found. Creating new log.
I20260619 12:38:46.192374 16818 log.cc:824] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:46.192374 16817 log.cc:824] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f: Log is configured to *not* fsync() on all Append() calls
I20260619 12:38:46.194450 16818 tablet_bootstrap.cc:492] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198: No bootstrap required, opened a new log
I20260619 12:38:46.194530 16817 tablet_bootstrap.cc:492] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f: No bootstrap required, opened a new log
I20260619 12:38:46.194675 16818 ts_tablet_manager.cc:1397] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260619 12:38:46.194720 16817 ts_tablet_manager.cc:1397] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260619 12:38:46.197723 16818 raft_consensus.cc:348] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:46.197754 16817 raft_consensus.cc:348] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:46.198040 16818 raft_consensus.cc:374] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:46.198074 16817 raft_consensus.cc:374] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:38:46.198134 16818 raft_consensus.cc:729] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c24392cb1a34dd48b840210f3b39198, State: Initialized, Role: FOLLOWER
I20260619 12:38:46.198168 16817 raft_consensus.cc:729] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb223adc7c8344a89e4a08ac34131a8f, State: Initialized, Role: FOLLOWER
I20260619 12:38:46.198666 16817 consensus_queue.cc:260] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:46.198673 16818 consensus_queue.cc:260] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:46.199409 16804 heartbeater.cc:503] Master 127.0.98.62:44289 was elected leader, sending a full tablet report...
I20260619 12:38:46.199409 16672 heartbeater.cc:503] Master 127.0.98.62:44289 was elected leader, sending a full tablet report...
I20260619 12:38:46.199834 16817 ts_tablet_manager.cc:1428] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260619 12:38:46.201289 16818 ts_tablet_manager.cc:1428] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
W20260619 12:38:46.213960 16535 tablet.cc:2369] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:38:46.333385 16540 raft_consensus.cc:482] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:38:46.333726 16540 raft_consensus.cc:504] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:46.334888 16540 leader_election.cc:288] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353), cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029)
I20260619 12:38:46.339248 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:46.335334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47017 (local address 127.0.98.2:38029)
0619 12:38:46.335614 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:46.335619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:46.335774 (+ 155us) server_negotiation.cc:407] Connection header received
0619 12:38:46.335940 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:46.335942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:46.335987 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:46.336070 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:46.336362 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.336873 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.338153 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.338576 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.338707 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:46.338777 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:46.339014 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 12:38:46.339101 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260619 12:38:46.339248 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:46.335423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44333 (local address 127.0.98.3:38353)
0619 12:38:46.335569 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:46.335573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:46.335686 (+ 113us) server_negotiation.cc:407] Connection header received
0619 12:38:46.335898 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:46.335901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:46.335953 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:46.336040 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:38:46.336364 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.336869 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.338144 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.338565 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.338685 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:46.338752 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:46.339011 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 12:38:46.339100 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 12:38:46.339282 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:46.335188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38353 (local address 127.0.98.1:44333)
0619 12:38:46.335577 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:46.335594 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:46.335757 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:46.336055 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:46.336058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:46.336075 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:46.336246 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:46.336251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.336986 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.336993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:46.338008 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:46.338018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.338694 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.338698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:46.338819 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:46.338899 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 12:38:46.338992 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 12:38:46.339100 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":73,"spinlock_wait_cycles":47360,"thread_start_us":124,"threads_started":1}
I20260619 12:38:46.339331 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:46.335188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38029 (local address 127.0.98.1:47017)
0619 12:38:46.335679 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:46.335690 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:46.335814 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:46.336089 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:38:46.336092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:38:46.336139 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:38:46.336245 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:46.336251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.337096 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.337100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:46.338016 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:38:46.338027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.338711 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.338715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:38:46.338870 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:46.338905 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 12:38:46.338998 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 12:38:46.339109 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":259,"spinlock_wait_cycles":768,"thread_start_us":50,"threads_started":1}
I20260619 12:38:46.340289 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d310b00f6c3e4259a2f33e36af4f047c" candidate_uuid: "3bcf34e242d1416abf64403baceba773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb223adc7c8344a89e4a08ac34131a8f" is_pre_election: true
I20260619 12:38:46.340292 16758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d310b00f6c3e4259a2f33e36af4f047c" candidate_uuid: "3bcf34e242d1416abf64403baceba773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c24392cb1a34dd48b840210f3b39198" is_pre_election: true
I20260619 12:38:46.340621 16758 raft_consensus.cc:2454] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bcf34e242d1416abf64403baceba773 in term 0.
I20260619 12:38:46.340628 16626 raft_consensus.cc:2454] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bcf34e242d1416abf64403baceba773 in term 0.
I20260619 12:38:46.341171 16423 leader_election.cc:302] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c24392cb1a34dd48b840210f3b39198, 3bcf34e242d1416abf64403baceba773; no voters:
I20260619 12:38:46.341539 16540 raft_consensus.cc:2781] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:38:46.341650 16540 raft_consensus.cc:482] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:38:46.341719 16540 raft_consensus.cc:3037] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:46.342962 16540 raft_consensus.cc:504] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:46.343418 16540 leader_election.cc:288] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 1 election: Requested vote from peers 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353), cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029)
I20260619 12:38:46.343883 16758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d310b00f6c3e4259a2f33e36af4f047c" candidate_uuid: "3bcf34e242d1416abf64403baceba773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c24392cb1a34dd48b840210f3b39198"
I20260619 12:38:46.343884 16626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d310b00f6c3e4259a2f33e36af4f047c" candidate_uuid: "3bcf34e242d1416abf64403baceba773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb223adc7c8344a89e4a08ac34131a8f"
I20260619 12:38:46.344023 16626 raft_consensus.cc:3037] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:46.344023 16758 raft_consensus.cc:3037] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:38:46.345397 16758 raft_consensus.cc:2454] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bcf34e242d1416abf64403baceba773 in term 1.
I20260619 12:38:46.345397 16626 raft_consensus.cc:2454] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bcf34e242d1416abf64403baceba773 in term 1.
I20260619 12:38:46.345752 16421 leader_election.cc:302] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bcf34e242d1416abf64403baceba773, cb223adc7c8344a89e4a08ac34131a8f; no voters:
I20260619 12:38:46.345963 16540 raft_consensus.cc:2781] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:38:46.346180 16540 raft_consensus.cc:686] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 LEADER]: Becoming Leader. State: Replica: 3bcf34e242d1416abf64403baceba773, State: Running, Role: LEADER
I20260619 12:38:46.346487 16540 consensus_queue.cc:237] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:38:46.349488 16342 catalog_manager.cc:5495] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bcf34e242d1416abf64403baceba773 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3bcf34e242d1416abf64403baceba773" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } health_report { overall_health: HEALTHY } } }
W20260619 12:38:46.361589 16805 tablet.cc:2369] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:38:46.372547 16827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:38:46.369048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46728 (local address 127.0.98.1:41777)
0619 12:38:46.369375 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0619 12:38:46.369383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:38:46.369397 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:38:46.369473 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:38:46.369475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:38:46.369528 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:38:46.369645 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:38:46.369893 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.370417 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.371184 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:38:46.371402 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:38:46.371512 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:38:46.371773 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:38:46.372213 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:38:46.372313 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 12:38:46.372364 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 12:38:46.372413 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":150,"threads_started":1}
I20260619 12:38:46.376519 16758 raft_consensus.cc:1261] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198 [term 1 FOLLOWER]: Refusing update from remote peer 3bcf34e242d1416abf64403baceba773: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:38:46.376519 16626 raft_consensus.cc:1261] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f [term 1 FOLLOWER]: Refusing update from remote peer 3bcf34e242d1416abf64403baceba773: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:38:46.377291 16540 consensus_queue.cc:1035] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:38:46.377558 16825 consensus_queue.cc:1035] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:38:46.386477 16830 mvcc.cc:205] Tried to move back new op lower bound from 7298550687231684608 to 7298550687117713408. Current Snapshot: MvccSnapshot[applied={T|T < 7298550687231684608}]
I20260619 12:38:46.386926 16829 mvcc.cc:205] Tried to move back new op lower bound from 7298550687231684608 to 7298550687117713408. Current Snapshot: MvccSnapshot[applied={T|T < 7298550687231684608}]
I20260619 12:38:46.396768 16346 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:38:46.396935 16346 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:38:46.399591 16346 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Sending DeleteTablet for 3 replicas of tablet d310b00f6c3e4259a2f33e36af4f047c
I20260619 12:38:46.400393 16468 tablet_service.cc:1514] Processing DeleteTablet for tablet d310b00f6c3e4259a2f33e36af4f047c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:38:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46722
I20260619 12:38:46.400503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 16384
I20260619 12:38:46.400420 16738 tablet_service.cc:1514] Processing DeleteTablet for tablet d310b00f6c3e4259a2f33e36af4f047c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:38:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56434
I20260619 12:38:46.401645 16606 tablet_service.cc:1514] Processing DeleteTablet for tablet d310b00f6c3e4259a2f33e36af4f047c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:38:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56824
I20260619 12:38:46.401754 16838 tablet_replica.cc:330] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198: stopping tablet replica
I20260619 12:38:46.402148 16839 tablet_replica.cc:330] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f: stopping tablet replica
I20260619 12:38:46.402158 16838 raft_consensus.cc:2229] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:38:46.402467 16839 raft_consensus.cc:2229] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:38:46.402524 16838 raft_consensus.cc:2258] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:38:46.402802 16839 raft_consensus.cc:2258] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:38:46.404565 16839 ts_tablet_manager.cc:1905] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:38:46.406898 16839 ts_tablet_manager.cc:1918] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:38:46.406996 16839 log.cc:1196] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/d310b00f6c3e4259a2f33e36af4f047c
I20260619 12:38:46.407315 16839 ts_tablet_manager.cc:1939] T d310b00f6c3e4259a2f33e36af4f047c P cb223adc7c8344a89e4a08ac34131a8f: Deleting consensus metadata
W20260619 12:38:46.408313 16331 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41777 (error 108)
I20260619 12:38:46.408849 16331 catalog_manager.cc:4841] TS cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029): tablet d310b00f6c3e4259a2f33e36af4f047c (table test-table [id=b5e5a1010ccb4d189701f9b8da33704b]) successfully deleted
W20260619 12:38:46.409509 16814 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:38:46.409144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41777 (local address 127.0.0.1:46738)
0619 12:38:46.409256 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 12:38:46.409270 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:38:46.409360 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:38:46.409433 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41777: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 12:38:46.409785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 16538
W20260619 12:38:46.409677 16331 catalog_manager.cc:4584] TS 3bcf34e242d1416abf64403baceba773 (127.0.98.1:41777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d310b00f6c3e4259a2f33e36af4f047c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41777: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260619 12:38:46.411818 16838 ts_tablet_manager.cc:1905] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:38:46.414098 16838 ts_tablet_manager.cc:1918] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:38:46.414197 16838 log.cc:1196] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/d310b00f6c3e4259a2f33e36af4f047c
I20260619 12:38:46.414510 16838 ts_tablet_manager.cc:1939] T d310b00f6c3e4259a2f33e36af4f047c P 1c24392cb1a34dd48b840210f3b39198: Deleting consensus metadata
I20260619 12:38:46.415787 16333 catalog_manager.cc:4841] TS 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353): tablet d310b00f6c3e4259a2f33e36af4f047c (table test-table [id=b5e5a1010ccb4d189701f9b8da33704b]) successfully deleted
I20260619 12:38:46.416077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 16675
I20260619 12:38:46.421782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 16313
I20260619 12:39:06.428637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44289
--webserver_interface=127.0.98.62
--webserver_port=32995
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:06.541682 16841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:06.542055 16841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:06.542137 16841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:06.546597 16841 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 12:39:06.546697 16841 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 12:39:06.546723 16841 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 12:39:06.546756 16841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:06.546798 16841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:06.546861 16841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:06.546904 16841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:06.546933 16841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:06.555110 16841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=32995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:06.556834 16841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:06.558112 16841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:39:06.567878 16841 server_base.cc:1034] running on GCE node
W20260619 12:39:06.567850 16846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:06.567925 16849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:06.567850 16847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:06.568740 16841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:06.569950 16841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:06.571141 16841 hybrid_clock.cc:648] HybridClock initialized: now 1781872746571104 us; error 55 us; skew 500 ppm
Jun 19 12:39:06 dist-test-slave-kdrb krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:06.578260 16841 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:39:06.579290 16841 webserver.cc:460] Webserver started at http://127.0.98.62:32995/ using document root <none> and password file <none>
I20260619 12:39:06.579932 16841 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:06.580000 16841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:06.584301 16841 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:39:06.586699 16856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:06.588228 16841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:06.588347 16841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "881a73a11f1e42ae88b903eb8cd8d379"
format_stamp: "Formatted at 2026-06-19 12:38:24 on dist-test-slave-kdrb"
server_key: "fe445a6b662e8527536b4e455a700e07"
server_key_iv: "8960d7126318e3441b3933e8297a9e11"
server_key_version: "encryptionkey@0"
I20260619 12:39:06.588922 16841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:06.608553 16841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:06.611670 16841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:06.611850 16841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:06.619549 16841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44289
I20260619 12:39:06.619839 16908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44289 every 8 connection(s)
I20260619 12:39:06.620836 16841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:06.625425 16909 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:39:06.626225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 16841
I20260619 12:39:06.626731 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41777
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44289
--webserver_port=35077
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:39:06.627548 16909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Bootstrap starting.
I20260619 12:39:06.646142 16909 log.cc:824] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:06.657972 16909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:39:06.658429 16909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Bootstrap complete.
I20260619 12:39:06.662657 16909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260619 12:39:06.663148 16909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881a73a11f1e42ae88b903eb8cd8d379, State: Initialized, Role: FOLLOWER
I20260619 12:39:06.663758 16909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260619 12:39:06.663930 16909 raft_consensus.cc:388] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:06.664016 16909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:06.664147 16909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 2 FOLLOWER]: Advancing to term 3
I20260619 12:39:06.666215 16909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260619 12:39:06.666594 16909 leader_election.cc:302] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 881a73a11f1e42ae88b903eb8cd8d379; no voters:
I20260619 12:39:06.666961 16909 leader_election.cc:288] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 12:39:06.667057 16913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 12:39:06.667254 16913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [term 3 LEADER]: Becoming Leader. State: Replica: 881a73a11f1e42ae88b903eb8cd8d379, State: Running, Role: LEADER
I20260619 12:39:06.667603 16913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260619 12:39:06.668368 16909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:06.669023 16914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "881a73a11f1e42ae88b903eb8cd8d379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } } }
I20260619 12:39:06.669170 16914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:06.670037 16919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:06.670233 16915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 881a73a11f1e42ae88b903eb8cd8d379. Latest consensus state: current_term: 3 leader_uuid: "881a73a11f1e42ae88b903eb8cd8d379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881a73a11f1e42ae88b903eb8cd8d379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } } }
I20260619 12:39:06.670414 16915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:06.675729 16919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:06.676577 16919 catalog_manager.cc:1259] Loaded cluster ID: 864809e86b524c18a0ecfcd4459678a7
I20260619 12:39:06.676649 16919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:06.679253 16919 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:06.680106 16919 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Loaded TSK: 0
I20260619 12:39:06.680202 16919 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 881a73a11f1e42ae88b903eb8cd8d379: Loaded TSK: 1
F20260619 12:39:06.681073 16919 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff133f099ad google::LogMessage::Fail() at ??:0
@ 0x7ff133f0c88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff133f094de google::LogMessage::Flush() at ??:0
@ 0x7ff133f0aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff136bcd18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff136bc5f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff13e86e7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff13e84899c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff13e849178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7ff13e88adf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff13e8a982b std::function<>::operator()() at ??:0
@ 0x7ff13e848b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ff13e849f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff13e846579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7ff13e88a640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff13615b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff13615bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff13615d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff13614c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff1377906db start_thread at ??:0
@ 0x7ff132a0761f clone at ??:0
W20260619 12:39:06.788511 16911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:06.788833 16911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:06.788884 16911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:06.793437 16911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:06.793509 16911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:06.793605 16911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:39:06.799280 16911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35077
--tserver_master_addrs=127.0.98.62:44289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:06.800516 16911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:06.801614 16911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:06.811863 16959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:06.811841 16956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:06.811869 16957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:06.812386 16911 server_base.cc:1034] running on GCE node
I20260619 12:39:06.812841 16911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:06.813561 16911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:06.814754 16911 hybrid_clock.cc:648] HybridClock initialized: now 1781872746814739 us; error 32 us; skew 500 ppm
Jun 19 12:39:06 dist-test-slave-kdrb krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:06.817796 16911 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:39:06.818796 16911 webserver.cc:460] Webserver started at http://127.0.98.1:35077/ using document root <none> and password file <none>
I20260619 12:39:06.819401 16911 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:06.819516 16911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:06.823942 16911 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 12:39:06.826433 16966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:06.827937 16911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:06.828101 16911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "3bcf34e242d1416abf64403baceba773"
format_stamp: "Formatted at 2026-06-19 12:38:24 on dist-test-slave-kdrb"
server_key: "2f0e6c2f87bd59d07e436dc9e733bdac"
server_key_iv: "a3cbdebfaf4392af9e770bfc1820f70f"
server_key_version: "encryptionkey@0"
I20260619 12:39:06.828702 16911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:06.861881 16911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:06.865228 16911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:06.865453 16911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:06.866067 16911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:06.867200 16973 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 12:39:06.872643 16911 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 12:39:06.872748 16911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 12:39:06.872835 16911 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 12:39:06.874575 16911 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 12:39:06.874648 16911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 12:39:06.874943 16973 tablet_bootstrap.cc:492] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773: Bootstrap starting.
I20260619 12:39:06.885723 16911 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41777
I20260619 12:39:06.885739 17080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41777 every 8 connection(s)
I20260619 12:39:06.886765 16911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:39:06.894815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 16911
I20260619 12:39:06.895326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38029
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44289
--webserver_port=42013
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:39:06.897747 16973 log.cc:824] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:06.907449 16973 tablet_bootstrap.cc:492] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:39:06.907963 16973 tablet_bootstrap.cc:492] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773: Bootstrap complete.
I20260619 12:39:06.909372 16973 ts_tablet_manager.cc:1397] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.010s
I20260619 12:39:06.913070 16973 raft_consensus.cc:348] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:39:06.913561 16973 raft_consensus.cc:729] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bcf34e242d1416abf64403baceba773, State: Initialized, Role: FOLLOWER
I20260619 12:39:06.914095 16973 consensus_queue.cc:260] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:39:06.915124 16973 ts_tablet_manager.cc:1428] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260619 12:39:06.917891 17083 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:39:06.889027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:55405)
0619 12:39:06.889552 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:06.889619 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:06.890477 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:06.917026 (+ 26549us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":150,"threads_started":1}
W20260619 12:39:06.918491 17081 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44289 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44289: Client connection negotiation failed: client connection to 127.0.98.62:44289: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 12:39:06.919247 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:06.918976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:36115)
0619 12:39:06.919102 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:06.919193 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 12:39:06.919998 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:06.919796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:56729)
0619 12:39:06.919900 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:06.919952 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 12:39:06.920192 17081 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:39:07.020745 17084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:07.021099 17084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:07.021188 17084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:07.026907 17084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:07.027015 17084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:07.027120 17084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:39:07.033305 17084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42013
--tserver_master_addrs=127.0.98.62:44289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:07.034469 17084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:07.035616 17084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:07.045933 17091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:07.045933 17092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:07.045950 17094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:07.046308 17084 server_base.cc:1034] running on GCE node
I20260619 12:39:07.046820 17084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:07.047616 17084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:07.048804 17084 hybrid_clock.cc:648] HybridClock initialized: now 1781872747048783 us; error 36 us; skew 500 ppm
Jun 19 12:39:07 dist-test-slave-kdrb krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:07.051702 17084 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:39:07.052771 17084 webserver.cc:460] Webserver started at http://127.0.98.2:42013/ using document root <none> and password file <none>
I20260619 12:39:07.053400 17084 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:07.053498 17084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:07.057673 17084 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 12:39:07.060086 17101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:07.061388 17084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 12:39:07.061537 17084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "cb223adc7c8344a89e4a08ac34131a8f"
format_stamp: "Formatted at 2026-06-19 12:38:24 on dist-test-slave-kdrb"
server_key: "9a250abf6166f45802ff7494254eebbe"
server_key_iv: "43307f89dd55ed5cfb37b82ac87395e9"
server_key_version: "encryptionkey@0"
I20260619 12:39:07.062142 17084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:07.075765 17084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:07.079150 17084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:07.079391 17084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:07.080078 17084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:07.080953 17084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:07.081040 17084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:07.081138 17084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:07.081190 17084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:07.090895 17084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38029
I20260619 12:39:07.090919 17214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38029 every 8 connection(s)
I20260619 12:39:07.092041 17084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:39:07.092878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 17084
I20260619 12:39:07.093369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38353
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44289
--webserver_port=35557
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:39:07.096803 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:07.094562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:51963)
0619 12:39:07.095143 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:07.095897 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":141,"threads_started":1}
W20260619 12:39:07.097322 17215 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44289 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44289: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
I20260619 12:39:07.098172 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:07.097833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:40675)
0619 12:39:07.097999 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:07.098070 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 12:39:07.098894 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:07.098622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:50053)
0619 12:39:07.098756 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:07.098818 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 12:39:07.099081 17215 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:39:07.211700 17217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:07.212023 17217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:07.212074 17217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:07.216269 17217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:07.216343 17217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:07.216408 17217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:39:07.222033 17217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35557
--tserver_master_addrs=127.0.98.62:44289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:07.223198 17217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:07.224395 17217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:07.234934 17226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:07.234949 17224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:07.234982 17223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:07.235165 17217 server_base.cc:1034] running on GCE node
I20260619 12:39:07.235675 17217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:07.236338 17217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:07.237511 17217 hybrid_clock.cc:648] HybridClock initialized: now 1781872747237491 us; error 34 us; skew 500 ppm
Jun 19 12:39:07 dist-test-slave-kdrb krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:07.240515 17217 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:39:07.241536 17217 webserver.cc:460] Webserver started at http://127.0.98.3:35557/ using document root <none> and password file <none>
I20260619 12:39:07.242158 17217 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:07.242261 17217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:07.246476 17217 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 12:39:07.248905 17233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:07.250264 17217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 12:39:07.250427 17217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "1c24392cb1a34dd48b840210f3b39198"
format_stamp: "Formatted at 2026-06-19 12:38:25 on dist-test-slave-kdrb"
server_key: "09bd69c7b9a2862cab33f7e4228028db"
server_key_iv: "ec7d9ba69084d0cebbc4b2a6b04af9c6"
server_key_version: "encryptionkey@0"
I20260619 12:39:07.251070 17217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:07.268980 17217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:07.272347 17217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:07.272575 17217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:07.273207 17217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:07.274099 17217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:07.274180 17217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:07.274283 17217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:07.274333 17217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:07.283602 17217 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38353
I20260619 12:39:07.283661 17346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38353 every 8 connection(s)
I20260619 12:39:07.284678 17217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:39:07.288928 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:07.286968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:44503)
0619 12:39:07.287503 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:07.287686 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":113,"threads_started":1}
W20260619 12:39:07.289506 17347 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44289 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44289: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
I20260619 12:39:07.290155 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:07.289903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:39715)
0619 12:39:07.290033 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:07.290094 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 12:39:07.290799 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:07.290591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:37623)
0619 12:39:07.290698 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:07.290751 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 12:39:07.290984 17347 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 12:39:07.291306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 17217
I20260619 12:39:07.921339 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:07.920886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:41877)
0619 12:39:07.921152 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:07.921216 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260619 12:39:08.078229 17352 raft_consensus.cc:482] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:39:08.078540 17352 raft_consensus.cc:504] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:39:08.079829 17352 leader_election.cc:288] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353), cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029)
Jun 19 12:39:08 dist-test-slave-kdrb krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 19 12:39:08 dist-test-slave-kdrb krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260619 12:39:08.095013 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:08.080092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38029 (local address 127.0.98.1:55079)
0619 12:39:08.080534 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:08.080548 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:08.080632 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:08.082822 (+ 2190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:08.082827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:08.082996 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:08.083292 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:08.083309 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:08.084760 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:08.084764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:08.085801 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:08.085812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:08.086561 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:08.086565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:08.086773 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:08.087584 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:39:08.087611 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:08.089416 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:08.092012 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:08.092018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:08.092032 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:08.092467 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:08.092843 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:08.092846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:08.092848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:08.093020 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:08.093917 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:08.093924 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:08.094245 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 12:39:08.094460 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 12:39:08.094820 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":98,"spinlock_wait_cycles":17792,"thread_start_us":80,"threads_started":1}
I20260619 12:39:08.095089 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:08.080166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49821 (local address 127.0.98.3:38353)
0619 12:39:08.080505 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:08.080542 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:08.080589 (+ 47us) server_negotiation.cc:407] Connection header received
0619 12:39:08.081486 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:08.081494 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:08.081981 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:08.082325 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:08.083586 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:08.084594 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:08.085962 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:08.086430 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:08.086612 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:08.089964 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:08.089969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:08.089981 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:08.090012 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:08.091960 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:08.092589 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:08.092599 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:08.092603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:08.092723 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:08.093112 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:08.093117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:08.093119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:08.093638 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:08.094022 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:08.094589 (+ 567us) server_negotiation.cc:299] Negotiation successful
0619 12:39:08.094894 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":113,"threads_started":1}
I20260619 12:39:08.095893 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:08.079993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38353 (local address 127.0.98.1:49821)
0619 12:39:08.080146 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:08.080162 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:08.080275 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:08.082404 (+ 2129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:08.082409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:08.082795 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:08.083295 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:08.083309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:08.084760 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:08.084769 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:08.085820 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:08.085831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:08.086563 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:08.086567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:08.086804 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:08.087611 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:39:08.087623 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:08.089774 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:08.092099 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:08.092102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:08.092105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:08.092455 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:08.092841 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:08.092844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:08.092846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:08.093013 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:08.093948 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:08.093951 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:08.094247 (+ 296us) client_negotiation.cc:769] Sending connection context
0619 12:39:08.094460 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 12:39:08.094910 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":387,"spinlock_wait_cycles":20480}
I20260619 12:39:08.095956 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:08.080361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55079 (local address 127.0.98.2:38029)
0619 12:39:08.080694 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:08.080706 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:08.080738 (+ 32us) server_negotiation.cc:407] Connection header received
0619 12:39:08.081485 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:08.081494 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:08.081981 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:08.082325 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:08.083518 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:08.084582 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:08.085937 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:08.086429 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:08.086612 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:08.089696 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:08.089704 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:08.089724 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:08.089760 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:08.091871 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:08.092593 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:08.092599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:08.092605 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:08.092737 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:08.093125 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:08.093128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:08.093130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:08.093659 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:08.093860 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:08.095462 (+ 1602us) server_negotiation.cc:299] Negotiation successful
0619 12:39:08.095766 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":125,"threads_started":1}
W20260619 12:39:08.098250 16970 leader_election.cc:341] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353): Not found: Tablet not found: d310b00f6c3e4259a2f33e36af4f047c
W20260619 12:39:08.098426 16968 leader_election.cc:341] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029): Not found: Tablet not found: d310b00f6c3e4259a2f33e36af4f047c
I20260619 12:39:08.098505 16968 leader_election.cc:302] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3bcf34e242d1416abf64403baceba773; no voters: 1c24392cb1a34dd48b840210f3b39198, cb223adc7c8344a89e4a08ac34131a8f
I20260619 12:39:08.098742 17352 raft_consensus.cc:2726] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:39:08.100070 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:08.099644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:45093)
0619 12:39:08.099901 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:08.099959 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260619 12:39:08.292102 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:08.291673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:48317)
0619 12:39:08.291945 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:08.292003 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
I20260619 12:39:08.922629 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:08.922184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:51751)
0619 12:39:08.922468 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:08.922531 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":113,"threads_started":1}
I20260619 12:39:09.101233 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:09.100826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:38639)
0619 12:39:09.101077 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:09.101138 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
I20260619 12:39:09.293465 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:09.292972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:56587)
0619 12:39:09.293263 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:09.293315 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":124,"threads_started":1}
I20260619 12:39:09.623267 17361 raft_consensus.cc:482] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:39:09.623452 17361 raft_consensus.cc:504] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:39:09.623982 17361 leader_election.cc:288] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353), cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029)
W20260619 12:39:09.624823 16970 leader_election.cc:341] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353): Not found: Tablet not found: d310b00f6c3e4259a2f33e36af4f047c
W20260619 12:39:09.624925 16968 leader_election.cc:341] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029): Not found: Tablet not found: d310b00f6c3e4259a2f33e36af4f047c
I20260619 12:39:09.625042 16968 leader_election.cc:302] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3bcf34e242d1416abf64403baceba773; no voters: 1c24392cb1a34dd48b840210f3b39198, cb223adc7c8344a89e4a08ac34131a8f
I20260619 12:39:09.625201 17361 raft_consensus.cc:2726] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:39:09.924048 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:09.923592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:49997)
0619 12:39:09.923879 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:09.923949 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":98,"threads_started":1}
I20260619 12:39:10.102586 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:10.102098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:47773)
0619 12:39:10.102404 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:10.102472 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":89,"threads_started":1}
I20260619 12:39:10.294780 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:10.294308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:43097)
0619 12:39:10.294591 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:10.294657 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
I20260619 12:39:10.925529 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:10.925035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:56873)
0619 12:39:10.925371 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:10.925427 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260619 12:39:11.103916 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:11.103432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:58095)
0619 12:39:11.103752 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:11.103815 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":144,"threads_started":1}
I20260619 12:39:11.209089 17367 raft_consensus.cc:482] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:39:11.209241 17367 raft_consensus.cc:504] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:39:11.209654 17367 leader_election.cc:288] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353), cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029)
W20260619 12:39:11.210681 16970 leader_election.cc:341] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353): Not found: Tablet not found: d310b00f6c3e4259a2f33e36af4f047c
W20260619 12:39:11.210825 16968 leader_election.cc:341] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029): Not found: Tablet not found: d310b00f6c3e4259a2f33e36af4f047c
I20260619 12:39:11.210896 16968 leader_election.cc:302] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3bcf34e242d1416abf64403baceba773; no voters: 1c24392cb1a34dd48b840210f3b39198, cb223adc7c8344a89e4a08ac34131a8f
I20260619 12:39:11.211047 17367 raft_consensus.cc:2726] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:39:11.296296 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:11.295808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:57355)
0619 12:39:11.296117 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:11.296190 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":93,"threads_started":1}
I20260619 12:39:11.927007 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:11.926504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:43967)
0619 12:39:11.926838 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:11.926907 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":74,"threads_started":1}
I20260619 12:39:12.105324 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:12.104872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:48441)
0619 12:39:12.105154 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:12.105222 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260619 12:39:12.297564 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:12.297116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:57829)
0619 12:39:12.297398 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:12.297462 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":91,"threads_started":1}
I20260619 12:39:12.928445 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:12.928012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:56977)
0619 12:39:12.928290 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:12.928350 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":91,"threads_started":1}
I20260619 12:39:13.106794 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:13.106259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:36771)
0619 12:39:13.106616 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:13.106694 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":90,"threads_started":1}
I20260619 12:39:13.298837 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:13.298404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:60839)
0619 12:39:13.298672 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:13.298725 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":105,"threads_started":1}
I20260619 12:39:13.534970 17375 raft_consensus.cc:482] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:39:13.535115 17375 raft_consensus.cc:504] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:39:13.535657 17375 leader_election.cc:288] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353), cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029)
W20260619 12:39:13.536516 16968 leader_election.cc:341] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029): Not found: Tablet not found: d310b00f6c3e4259a2f33e36af4f047c
W20260619 12:39:13.536619 16970 leader_election.cc:341] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353): Not found: Tablet not found: d310b00f6c3e4259a2f33e36af4f047c
I20260619 12:39:13.536685 16970 leader_election.cc:302] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3bcf34e242d1416abf64403baceba773; no voters: 1c24392cb1a34dd48b840210f3b39198, cb223adc7c8344a89e4a08ac34131a8f
I20260619 12:39:13.536852 17375 raft_consensus.cc:2726] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:39:13.929684 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:13.929243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:60119)
0619 12:39:13.929508 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:13.929583 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260619 12:39:14.108129 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:14.107635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:39747)
0619 12:39:14.107947 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:14.108028 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":86,"threads_started":1}
I20260619 12:39:14.300313 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:14.299732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:54991)
0619 12:39:14.300143 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:14.300213 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":115,"threads_started":1}
I20260619 12:39:14.931254 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:14.930708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:48485)
0619 12:39:14.931060 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:14.931149 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
I20260619 12:39:15.109606 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:15.109061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:33275)
0619 12:39:15.109402 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:15.109476 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":83,"threads_started":1}
I20260619 12:39:15.301573 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:15.301138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:48019)
0619 12:39:15.301405 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:15.301469 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260619 12:39:15.932695 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:15.932197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:42869)
0619 12:39:15.932502 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:15.932563 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":87,"threads_started":1}
I20260619 12:39:16.110987 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:16.110548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:56365)
0619 12:39:16.110819 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:16.110886 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20260619 12:39:16.302953 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:16.302487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:58091)
0619 12:39:16.302781 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:16.302848 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
I20260619 12:39:16.934099 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:16.933607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:37223)
0619 12:39:16.933905 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:16.933977 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
I20260619 12:39:17.112318 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:17.111878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:40623)
0619 12:39:17.112144 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:17.112203 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":90,"threads_started":1}
I20260619 12:39:17.304490 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:17.303994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:36841)
0619 12:39:17.304315 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:17.304384 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":96,"threads_started":1}
I20260619 12:39:17.387931 17388 raft_consensus.cc:482] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:39:17.388094 17388 raft_consensus.cc:504] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c24392cb1a34dd48b840210f3b39198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38353 } } peers { permanent_uuid: "cb223adc7c8344a89e4a08ac34131a8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38029 } } peers { permanent_uuid: "3bcf34e242d1416abf64403baceba773" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } }
I20260619 12:39:17.388522 17388 leader_election.cc:288] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353), cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029)
W20260619 12:39:17.389417 16970 leader_election.cc:341] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c24392cb1a34dd48b840210f3b39198 (127.0.98.3:38353): Not found: Tablet not found: d310b00f6c3e4259a2f33e36af4f047c
W20260619 12:39:17.389533 16968 leader_election.cc:341] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb223adc7c8344a89e4a08ac34131a8f (127.0.98.2:38029): Not found: Tablet not found: d310b00f6c3e4259a2f33e36af4f047c
I20260619 12:39:17.389597 16968 leader_election.cc:302] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3bcf34e242d1416abf64403baceba773; no voters: 1c24392cb1a34dd48b840210f3b39198, cb223adc7c8344a89e4a08ac34131a8f
I20260619 12:39:17.389735 17388 raft_consensus.cc:2726] T d310b00f6c3e4259a2f33e36af4f047c P 3bcf34e242d1416abf64403baceba773 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:39:17.935426 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:17.934957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:39461)
0619 12:39:17.935261 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:17.935327 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":80,"threads_started":1}
I20260619 12:39:18.113718 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:18.113229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:43715)
0619 12:39:18.113546 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:18.113615 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":85,"threads_started":1}
I20260619 12:39:18.305775 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:18.305297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:47915)
0619 12:39:18.305585 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:18.305673 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
I20260619 12:39:18.936744 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:18.936319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:59089)
0619 12:39:18.936590 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:18.936646 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":94,"threads_started":1}
I20260619 12:39:19.115082 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:19.114557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:43195)
0619 12:39:19.114907 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:19.114971 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":99,"threads_started":1}
I20260619 12:39:19.307137 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:19.306668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:44577)
0619 12:39:19.306948 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:19.307013 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":97,"threads_started":1}
I20260619 12:39:19.938115 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:19.937638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:47931)
0619 12:39:19.937938 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:19.938011 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":83,"threads_started":1}
I20260619 12:39:20.116428 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:20.115984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:57145)
0619 12:39:20.116262 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:20.116329 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260619 12:39:20.308429 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:20.307967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:39643)
0619 12:39:20.308268 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:20.308326 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":93,"threads_started":1}
I20260619 12:39:20.939354 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:20.938913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:53593)
0619 12:39:20.939190 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:20.939257 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
I20260619 12:39:21.117815 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:21.117357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:41987)
0619 12:39:21.117645 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:21.117711 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":82,"threads_started":1}
I20260619 12:39:21.309929 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:21.309386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:58355)
0619 12:39:21.309730 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:21.309821 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":99,"threads_started":1}
I20260619 12:39:21.940909 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:21.940397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:39021)
0619 12:39:21.940729 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:21.940806 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":74,"threads_started":1}
I20260619 12:39:22.119305 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:22.118843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:34775)
0619 12:39:22.119140 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:22.119209 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":76,"threads_started":1}
I20260619 12:39:22.297922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 16911
I20260619 12:39:22.304041 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 17084
I20260619 12:39:22.309636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 17217
2026-06-19T12:39:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59277 ms)
[----------] 30 tests from SecurityITest (245330 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:22 dist-test-slave-kdrb krb5kdc[17405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:22 dist-test-slave-kdrb krb5kdc[17405](info): set up 2 sockets
Jun 19 12:39:22 dist-test-slave-kdrb krb5kdc[17405](info): commencing operation
krb5kdc: starting...
W20260619 12:39:23.355522 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:23 dist-test-slave-kdrb krb5kdc[17405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:39:23.479182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39647
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:23.594223 17418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:23.594532 17418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:23.594579 17418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:23.598796 17418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:23.598875 17418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:23.598896 17418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:23.598912 17418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:23.598927 17418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:23.604434 17418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:23.605675 17418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:23.606712 17418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:23.615823 17424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:23.615799 17423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:23.615799 17426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:23.616255 17418 server_base.cc:1034] running on GCE node
I20260619 12:39:23.616905 17418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:23.618156 17418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:23.619387 17418 hybrid_clock.cc:648] HybridClock initialized: now 1781872763619372 us; error 30 us; skew 500 ppm
Jun 19 12:39:23 dist-test-slave-kdrb krb5kdc[17405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:23.631592 17418 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:39:23.632911 17418 webserver.cc:460] Webserver started at http://127.0.98.62:41071/ using document root <none> and password file <none>
I20260619 12:39:23.633562 17418 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:23.633666 17418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:23.633931 17418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:23.636332 17418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "ab75f7a2a4154e81adacad74471dbf5d"
format_stamp: "Formatted at 2026-06-19 12:39:23 on dist-test-slave-kdrb"
server_key: "95464cc8859e7248fc3074813d49362a"
server_key_iv: "dca7154bf38164cc7e7f272417257bcf"
server_key_version: "encryptionkey@0"
I20260619 12:39:23.636963 17418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "ab75f7a2a4154e81adacad74471dbf5d"
format_stamp: "Formatted at 2026-06-19 12:39:23 on dist-test-slave-kdrb"
server_key: "95464cc8859e7248fc3074813d49362a"
server_key_iv: "dca7154bf38164cc7e7f272417257bcf"
server_key_version: "encryptionkey@0"
I20260619 12:39:23.641331 17418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:39:23.644337 17433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:23.645844 17418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 12:39:23.646008 17418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "ab75f7a2a4154e81adacad74471dbf5d"
format_stamp: "Formatted at 2026-06-19 12:39:23 on dist-test-slave-kdrb"
server_key: "95464cc8859e7248fc3074813d49362a"
server_key_iv: "dca7154bf38164cc7e7f272417257bcf"
server_key_version: "encryptionkey@0"
I20260619 12:39:23.646143 17418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:23.669143 17418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:23.672734 17418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:23.672969 17418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:23.680966 17418 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39647
I20260619 12:39:23.680994 17485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39647 every 8 connection(s)
I20260619 12:39:23.682021 17418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:23.684958 17486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:23.686462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 17418
I20260619 12:39:23.686576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:23.686894 392 external_mini_cluster.cc:1428] Setting key bf6c66e2afb45862d61a5eab17631c00
I20260619 12:39:23.691373 17486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d: Bootstrap starting.
I20260619 12:39:23.693830 17486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:23.694615 17486 log.cc:824] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:39:23 dist-test-slave-kdrb krb5kdc[17405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:39:23.696745 17486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d: No bootstrap required, opened a new log
I20260619 12:39:23.700461 17486 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab75f7a2a4154e81adacad74471dbf5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39647 } }
I20260619 12:39:23.700786 17486 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:23.700873 17486 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab75f7a2a4154e81adacad74471dbf5d, State: Initialized, Role: FOLLOWER
I20260619 12:39:23.701400 17486 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab75f7a2a4154e81adacad74471dbf5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39647 } }
I20260619 12:39:23.701473 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:23.688427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48764 (local address 127.0.98.62:39647)
0619 12:39:23.688921 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:23.688937 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:23.688976 (+ 39us) server_negotiation.cc:407] Connection header received
0619 12:39:23.689626 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:23.689643 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:23.690018 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:23.690384 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:23.691222 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:23.692344 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:23.693226 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:23.693724 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:23.693904 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:23.696529 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:23.696534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:23.696549 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:23.696577 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:23.698348 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:23.698833 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:23.698838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:23.698843 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:23.698961 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:23.699254 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:23.699258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:23.699260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:23.699755 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:23.699985 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:23.700231 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 12:39:23.700553 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":157,"threads_started":1}
I20260619 12:39:23.701579 17486 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:23.701654 17486 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:23.701781 17486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:23.702999 17486 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab75f7a2a4154e81adacad74471dbf5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39647 } }
I20260619 12:39:23.703385 17486 leader_election.cc:302] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab75f7a2a4154e81adacad74471dbf5d; no voters:
I20260619 12:39:23.703780 17486 leader_election.cc:288] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:23.703862 17491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:23.704070 17491 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [term 1 LEADER]: Becoming Leader. State: Replica: ab75f7a2a4154e81adacad74471dbf5d, State: Running, Role: LEADER
I20260619 12:39:23.704442 17491 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab75f7a2a4154e81adacad74471dbf5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39647 } }
I20260619 12:39:23.705193 17486 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:23.706341 17492 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab75f7a2a4154e81adacad74471dbf5d. Latest consensus state: current_term: 1 leader_uuid: "ab75f7a2a4154e81adacad74471dbf5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab75f7a2a4154e81adacad74471dbf5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39647 } } }
I20260619 12:39:23.706496 17492 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:23.706751 17493 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab75f7a2a4154e81adacad74471dbf5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab75f7a2a4154e81adacad74471dbf5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39647 } } }
I20260619 12:39:23.706841 17493 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:23.706921 17500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:23.709882 17500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:23.716048 17500 catalog_manager.cc:1347] Generated new cluster ID: 5f402d3769614dbaa5419074e24488f0
I20260619 12:39:23.716138 17500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:23.722219 17500 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:23.723703 17500 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:23.737736 17500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d: Generated new TSK 0
I20260619 12:39:23.738572 17500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:39:23.774457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39647
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:23.896588 17512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:23.896900 17512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:23.896944 17512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:23.901234 17512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:23.901314 17512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:23.901376 17512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:39:23.907387 17512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:23.908617 17512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:23.909732 17512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:23.919967 17517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:23.920022 17512 server_base.cc:1034] running on GCE node
W20260619 12:39:23.919948 17518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:23.919948 17520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:23.920825 17512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:23.921520 17512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:23.922717 17512 hybrid_clock.cc:648] HybridClock initialized: now 1781872763922696 us; error 34 us; skew 500 ppm
Jun 19 12:39:23 dist-test-slave-kdrb krb5kdc[17405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:23.925843 17512 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:39:23.926836 17512 webserver.cc:460] Webserver started at http://127.0.98.1:42883/ using document root <none> and password file <none>
I20260619 12:39:23.927448 17512 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:23.927575 17512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:23.927806 17512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:23.929989 17512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "ec6c330979de44deaeaa07281758d659"
format_stamp: "Formatted at 2026-06-19 12:39:23 on dist-test-slave-kdrb"
server_key: "d743cb0e0d4c551845e23e6cccd13fa7"
server_key_iv: "b11ddb88b01915039ad26dd8431cb769"
server_key_version: "encryptionkey@0"
I20260619 12:39:23.930653 17512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ec6c330979de44deaeaa07281758d659"
format_stamp: "Formatted at 2026-06-19 12:39:23 on dist-test-slave-kdrb"
server_key: "d743cb0e0d4c551845e23e6cccd13fa7"
server_key_iv: "b11ddb88b01915039ad26dd8431cb769"
server_key_version: "encryptionkey@0"
I20260619 12:39:23.935143 17512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:39:23.937865 17527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:23.939200 17512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:39:23.939344 17512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "ec6c330979de44deaeaa07281758d659"
format_stamp: "Formatted at 2026-06-19 12:39:23 on dist-test-slave-kdrb"
server_key: "d743cb0e0d4c551845e23e6cccd13fa7"
server_key_iv: "b11ddb88b01915039ad26dd8431cb769"
server_key_version: "encryptionkey@0"
I20260619 12:39:23.939515 17512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:23.952557 17512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:23.955716 17512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:23.955929 17512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:23.956593 17512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:23.957444 17512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:23.957491 17512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:23.957561 17512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:23.957592 17512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:23.966725 17512 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33255
I20260619 12:39:23.966755 17640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33255 every 8 connection(s)
I20260619 12:39:23.967797 17512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:39:23.971698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 17512
I20260619 12:39:23.971868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:39:23.972188 392 external_mini_cluster.cc:1428] Setting key fd69e12427667f326fc81446e6fb158d
Jun 19 12:39:23 dist-test-slave-kdrb krb5kdc[17405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:39:23.983316 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:23.969659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38943 (local address 127.0.98.62:39647)
0619 12:39:23.969813 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:23.969817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:23.970674 (+ 857us) server_negotiation.cc:407] Connection header received
0619 12:39:23.972133 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:23.972138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:23.972195 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:23.972325 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:23.973935 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:23.974734 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:23.976011 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:23.976416 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:23.976548 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:23.979512 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:23.979516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:23.979519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:23.979554 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:23.981040 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:23.981583 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:23.981588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:23.981590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:23.981670 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:23.982091 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:23.982095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:23.982097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:23.982296 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:23.982402 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:23.982985 (+ 583us) server_negotiation.cc:299] Negotiation successful
0619 12:39:23.983122 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 12:39:23.984354 17643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:23.970064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39647 (local address 127.0.98.1:38943)
0619 12:39:23.970507 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:23.970555 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:23.971857 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:23.972507 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:23.972514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:23.973033 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:23.973764 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:23.973790 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:23.974859 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:23.974862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:23.975884 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:23.975893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:23.976585 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:23.976590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:23.976829 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:23.977512 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:39:23.977538 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:23.979294 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:23.981172 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:23.981179 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:23.981197 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:23.981475 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:23.981766 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:23.981769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:23.981771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:23.981989 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:23.982417 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:23.982426 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:23.982696 (+ 270us) client_negotiation.cc:769] Sending connection context
0619 12:39:23.982900 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 12:39:23.983256 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:39:23.985769 17641 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39647
I20260619 12:39:23.986060 17641 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:23.986676 17641 heartbeater.cc:511] Master 127.0.98.62:39647 requested a full tablet report, sending...
I20260619 12:39:23.988384 17450 ts_manager.cc:194] Registered new tserver with Master: ec6c330979de44deaeaa07281758d659 (127.0.98.1:33255)
I20260619 12:39:23.989609 17450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:39:24.003806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39647
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:24.123060 17646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:24.123360 17646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:24.123406 17646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:24.127676 17646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:24.127746 17646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:24.127816 17646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:39:24.133723 17646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:24.134832 17646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:24.135926 17646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:39:24.146358 17646 server_base.cc:1034] running on GCE node
W20260619 12:39:24.146310 17652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:24.146382 17651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:24.146310 17654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:24.147044 17646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:24.147804 17646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:24.148998 17646 hybrid_clock.cc:648] HybridClock initialized: now 1781872764148981 us; error 32 us; skew 500 ppm
Jun 19 12:39:24 dist-test-slave-kdrb krb5kdc[17405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:24.152019 17646 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:39:24.153014 17646 webserver.cc:460] Webserver started at http://127.0.98.2:43637/ using document root <none> and password file <none>
I20260619 12:39:24.153601 17646 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:24.153690 17646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:24.153923 17646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:24.156121 17646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "cb3c446f02494636b9c4d80ab500cc3b"
format_stamp: "Formatted at 2026-06-19 12:39:24 on dist-test-slave-kdrb"
server_key: "b87bfb8b444d7c48f95ae5971c5c53b0"
server_key_iv: "85c99837d30a44dfc7c4cde82a8f3d76"
server_key_version: "encryptionkey@0"
I20260619 12:39:24.156716 17646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cb3c446f02494636b9c4d80ab500cc3b"
format_stamp: "Formatted at 2026-06-19 12:39:24 on dist-test-slave-kdrb"
server_key: "b87bfb8b444d7c48f95ae5971c5c53b0"
server_key_iv: "85c99837d30a44dfc7c4cde82a8f3d76"
server_key_version: "encryptionkey@0"
I20260619 12:39:24.160673 17646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:39:24.163332 17661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:24.164814 17646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:24.164966 17646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "cb3c446f02494636b9c4d80ab500cc3b"
format_stamp: "Formatted at 2026-06-19 12:39:24 on dist-test-slave-kdrb"
server_key: "b87bfb8b444d7c48f95ae5971c5c53b0"
server_key_iv: "85c99837d30a44dfc7c4cde82a8f3d76"
server_key_version: "encryptionkey@0"
I20260619 12:39:24.165093 17646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:24.181946 17646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:24.185379 17646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:24.185617 17646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:24.186427 17646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:24.187356 17646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:24.187433 17646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:24.187530 17646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:24.187573 17646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:24.197220 17646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45895
I20260619 12:39:24.197244 17774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45895 every 8 connection(s)
I20260619 12:39:24.198199 17646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:39:24.201323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 17646
I20260619 12:39:24.201445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:39:24.201732 392 external_mini_cluster.cc:1428] Setting key 9251d1a16e675662d370cfbd3676799a
Jun 19 12:39:24 dist-test-slave-kdrb krb5kdc[17405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:39:24.214555 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.200153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34355 (local address 127.0.98.62:39647)
0619 12:39:24.200337 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:24.200342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:24.201364 (+ 1022us) server_negotiation.cc:407] Connection header received
0619 12:39:24.202883 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:24.202886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:24.202941 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:24.203025 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:24.204943 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.205582 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.207224 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.207577 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.207697 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.210697 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:24.210701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:24.210703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:24.210730 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:24.212344 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:24.212945 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:24.212949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:24.212951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:24.213038 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:24.213445 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:24.213448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:24.213450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:24.213666 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:24.213759 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:24.214306 (+ 547us) server_negotiation.cc:299] Negotiation successful
0619 12:39:24.214428 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 12:39:24.215425 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.200628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39647 (local address 127.0.98.2:34355)
0619 12:39:24.201192 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:24.201241 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:24.202647 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:24.203519 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:24.203535 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:24.204108 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:24.204778 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:24.204794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.205768 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.205775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:24.207105 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:24.207116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.207764 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.207769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:24.207953 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.208659 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:39:24.208678 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:24.210493 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:24.212491 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:24.212498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:24.212519 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:24.212830 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:24.213170 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:24.213173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:24.213174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:24.213351 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:24.213772 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:24.213777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:24.214052 (+ 275us) client_negotiation.cc:769] Sending connection context
0619 12:39:24.214221 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 12:39:24.214522 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:39:24.216845 17775 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39647
I20260619 12:39:24.217106 17775 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:24.217628 17775 heartbeater.cc:511] Master 127.0.98.62:39647 requested a full tablet report, sending...
I20260619 12:39:24.218786 17450 ts_manager.cc:194] Registered new tserver with Master: cb3c446f02494636b9c4d80ab500cc3b (127.0.98.2:45895)
I20260619 12:39:24.219717 17450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:39:24.235303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39647
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:24.350678 17780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:24.350978 17780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:24.351058 17780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:24.355307 17780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:24.355393 17780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:24.355458 17780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:39:24.361152 17780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:24.362339 17780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:24.363422 17780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:24.373749 17785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:24.373749 17788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:24.373749 17786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:24.373790 17780 server_base.cc:1034] running on GCE node
I20260619 12:39:24.374467 17780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:24.375092 17780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:24.376298 17780 hybrid_clock.cc:648] HybridClock initialized: now 1781872764376285 us; error 39 us; skew 500 ppm
Jun 19 12:39:24 dist-test-slave-kdrb krb5kdc[17405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:24.379132 17780 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:39:24.380203 17780 webserver.cc:460] Webserver started at http://127.0.98.3:42271/ using document root <none> and password file <none>
I20260619 12:39:24.380833 17780 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:24.380935 17780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:24.381186 17780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:24.383409 17780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "ee52c09beab44ec2a44279a09252a8fe"
format_stamp: "Formatted at 2026-06-19 12:39:24 on dist-test-slave-kdrb"
server_key: "49165befe25aa9e72e8399911ab76bca"
server_key_iv: "8c1f9077025479842b7177cc0d94bb3e"
server_key_version: "encryptionkey@0"
I20260619 12:39:24.384106 17780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ee52c09beab44ec2a44279a09252a8fe"
format_stamp: "Formatted at 2026-06-19 12:39:24 on dist-test-slave-kdrb"
server_key: "49165befe25aa9e72e8399911ab76bca"
server_key_iv: "8c1f9077025479842b7177cc0d94bb3e"
server_key_version: "encryptionkey@0"
I20260619 12:39:24.388217 17780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:39:24.390965 17795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:24.392457 17780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:24.392616 17780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "ee52c09beab44ec2a44279a09252a8fe"
format_stamp: "Formatted at 2026-06-19 12:39:24 on dist-test-slave-kdrb"
server_key: "49165befe25aa9e72e8399911ab76bca"
server_key_iv: "8c1f9077025479842b7177cc0d94bb3e"
server_key_version: "encryptionkey@0"
I20260619 12:39:24.392763 17780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:24.412297 17780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:24.415908 17780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:24.416136 17780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:24.416783 17780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:24.417665 17780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:24.417737 17780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:24.417804 17780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:24.417848 17780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:24.427711 17780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39591
I20260619 12:39:24.427731 17908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39591 every 8 connection(s)
I20260619 12:39:24.428699 17780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:39:24.432168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 17780
I20260619 12:39:24.432361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:39:24.432739 392 external_mini_cluster.cc:1428] Setting key 633c71c5c87083cd04a9b3bb309d41e0
Jun 19 12:39:24 dist-test-slave-kdrb krb5kdc[17405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:39:24.443562 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.430593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44851 (local address 127.0.98.62:39647)
0619 12:39:24.430749 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:24.430753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:24.431751 (+ 998us) server_negotiation.cc:407] Connection header received
0619 12:39:24.433503 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:24.433507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:24.433566 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:24.433673 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:24.435082 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.435686 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.436850 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.437074 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.437188 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.439799 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:24.439802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:24.439804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:24.439831 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:24.441346 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:24.441917 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:24.441920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:24.441921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:24.441997 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:24.442360 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:24.442364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:24.442365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:24.442622 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:24.442739 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:24.443264 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 12:39:24.443383 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 12:39:24.444489 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.431029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39647 (local address 127.0.98.3:44851)
0619 12:39:24.431575 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:24.431639 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:24.433265 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:24.433876 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:24.433885 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:24.434299 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:24.434938 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:24.434954 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.435821 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.435824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:24.436714 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:24.436723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.437229 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.437234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:24.437434 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.438080 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:39:24.438097 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:24.439609 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:24.441485 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:24.441491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:24.441509 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:24.441791 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:24.442090 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:24.442094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:24.442096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:24.442256 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:24.442759 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:24.442765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:24.443035 (+ 270us) client_negotiation.cc:769] Sending connection context
0619 12:39:24.443215 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 12:39:24.443573 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260619 12:39:24.445794 17909 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39647
I20260619 12:39:24.446084 17909 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:24.446708 17909 heartbeater.cc:511] Master 127.0.98.62:39647 requested a full tablet report, sending...
I20260619 12:39:24.447814 17450 ts_manager.cc:194] Registered new tserver with Master: ee52c09beab44ec2a44279a09252a8fe (127.0.98.3:39591)
I20260619 12:39:24.448402 17450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44851
I20260619 12:39:24.458897 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:39:24.468317 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.460999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.0.98.62:39647)
0619 12:39:24.461125 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:24.461129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:24.461318 (+ 189us) server_negotiation.cc:407] Connection header received
0619 12:39:24.461414 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:24.461416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:24.461457 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:24.461527 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:24.462210 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.462746 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.463658 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.463867 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.463948 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.465354 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:24.465357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:24.465359 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:24.465382 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:24.466662 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:24.467062 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:24.467065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:24.467066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:24.467149 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:24.467409 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:24.467412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:24.467414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:24.467701 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:24.467821 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:24.468061 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 12:39:24.468169 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:39:24.471467 17450 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:39:24.474596 17450 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:39:24.489991 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.483843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.0.98.3:39591)
0619 12:39:24.484396 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:24.484411 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:24.484436 (+ 25us) server_negotiation.cc:407] Connection header received
0619 12:39:24.484762 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:24.484774 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:24.485016 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:24.485185 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:24.485686 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.486872 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.488408 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.489253 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.489373 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.489548 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:24.489698 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 12:39:24.489812 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":180,"threads_started":1}
I20260619 12:39:24.489985 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.484529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45895 (local address 127.0.0.1:39402)
0619 12:39:24.484855 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:24.484868 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:24.484950 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:24.485868 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:24.485871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:24.485888 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:24.485989 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:24.485995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.487062 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.487067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:24.488487 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:24.488496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.489534 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.489538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:24.489674 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.489698 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 12:39:24.489749 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 12:39:24.489822 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":82,"thread_start_us":94,"threads_started":1}
I20260619 12:39:24.489997 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.483701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39591 (local address 127.0.0.1:39700)
0619 12:39:24.484293 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:24.484348 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:24.484495 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:24.485396 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:24.485402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:24.485428 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:24.485561 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:24.485568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.487028 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.487032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:24.488274 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:24.488289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.489381 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.489385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:24.489489 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.489533 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 12:39:24.489668 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 12:39:24.489818 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":2,"thread_start_us":121,"threads_started":1}
I20260619 12:39:24.490201 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.484866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33255 (local address 127.0.0.1:52842)
0619 12:39:24.485161 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:24.485176 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:24.485265 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:24.486026 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:24.486030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:24.486048 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:24.486137 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:24.486143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.487840 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.487843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:24.488853 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:24.488860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.489781 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.489785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:24.489862 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.489884 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:39:24.489933 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 12:39:24.489998 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":80,"threads_started":1}
I20260619 12:39:24.490765 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.485261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.0.98.1:33255)
0619 12:39:24.485549 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:24.485562 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:24.485581 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:39:24.485670 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:24.485675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:24.485870 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:24.486281 (+ 411us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:24.486408 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.487722 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.488967 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.489658 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.490289 (+ 631us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.490452 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:24.490501 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 12:39:24.490651 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":118,"threads_started":1}
I20260619 12:39:24.490978 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.484758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.0.98.2:45895)
0619 12:39:24.485402 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:24.485415 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:24.485437 (+ 22us) server_negotiation.cc:407] Connection header received
0619 12:39:24.485508 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:24.485512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:24.485704 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:24.485886 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:24.486101 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.486904 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.488618 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.489399 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.490524 (+ 1125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.490673 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:24.490721 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 12:39:24.490820 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":110,"threads_started":1}
I20260619 12:39:24.493520 17575 tablet_service.cc:1467] Processing CreateTablet for tablet 21be0899a1af45f8895e54a997c4f6cc (DEFAULT_TABLE table=test-table [id=3982041207ac4a2d8eb5ea2f4cdaf43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:24.493646 17709 tablet_service.cc:1467] Processing CreateTablet for tablet 21be0899a1af45f8895e54a997c4f6cc (DEFAULT_TABLE table=test-table [id=3982041207ac4a2d8eb5ea2f4cdaf43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:24.493644 17843 tablet_service.cc:1467] Processing CreateTablet for tablet 21be0899a1af45f8895e54a997c4f6cc (DEFAULT_TABLE table=test-table [id=3982041207ac4a2d8eb5ea2f4cdaf43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:24.494669 17843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21be0899a1af45f8895e54a997c4f6cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:24.494669 17709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21be0899a1af45f8895e54a997c4f6cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:24.494670 17575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21be0899a1af45f8895e54a997c4f6cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:24.501572 17926 tablet_bootstrap.cc:492] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659: Bootstrap starting.
I20260619 12:39:24.502737 17927 tablet_bootstrap.cc:492] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe: Bootstrap starting.
I20260619 12:39:24.502737 17928 tablet_bootstrap.cc:492] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b: Bootstrap starting.
I20260619 12:39:24.504319 17926 tablet_bootstrap.cc:654] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:24.505081 17927 tablet_bootstrap.cc:654] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:24.505074 17928 tablet_bootstrap.cc:654] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:24.505622 17926 log.cc:824] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:24.506409 17927 log.cc:824] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:24.506408 17928 log.cc:824] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:24.508108 17926 tablet_bootstrap.cc:492] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659: No bootstrap required, opened a new log
I20260619 12:39:24.508347 17926 ts_tablet_manager.cc:1397] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260619 12:39:24.508420 17927 tablet_bootstrap.cc:492] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe: No bootstrap required, opened a new log
I20260619 12:39:24.508618 17928 tablet_bootstrap.cc:492] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b: No bootstrap required, opened a new log
I20260619 12:39:24.508606 17927 ts_tablet_manager.cc:1397] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 12:39:24.508781 17928 ts_tablet_manager.cc:1397] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 12:39:24.511942 17926 raft_consensus.cc:348] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee52c09beab44ec2a44279a09252a8fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } } peers { permanent_uuid: "cb3c446f02494636b9c4d80ab500cc3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "ec6c330979de44deaeaa07281758d659" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260619 12:39:24.512198 17926 raft_consensus.cc:374] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:24.512250 17926 raft_consensus.cc:729] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6c330979de44deaeaa07281758d659, State: Initialized, Role: FOLLOWER
I20260619 12:39:24.512563 17928 raft_consensus.cc:348] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee52c09beab44ec2a44279a09252a8fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } } peers { permanent_uuid: "cb3c446f02494636b9c4d80ab500cc3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "ec6c330979de44deaeaa07281758d659" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260619 12:39:24.512614 17927 raft_consensus.cc:348] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee52c09beab44ec2a44279a09252a8fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } } peers { permanent_uuid: "cb3c446f02494636b9c4d80ab500cc3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "ec6c330979de44deaeaa07281758d659" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260619 12:39:24.512734 17926 consensus_queue.cc:260] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee52c09beab44ec2a44279a09252a8fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } } peers { permanent_uuid: "cb3c446f02494636b9c4d80ab500cc3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "ec6c330979de44deaeaa07281758d659" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260619 12:39:24.512838 17927 raft_consensus.cc:374] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:24.512840 17928 raft_consensus.cc:374] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:24.512889 17927 raft_consensus.cc:729] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee52c09beab44ec2a44279a09252a8fe, State: Initialized, Role: FOLLOWER
I20260619 12:39:24.512889 17928 raft_consensus.cc:729] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3c446f02494636b9c4d80ab500cc3b, State: Initialized, Role: FOLLOWER
I20260619 12:39:24.513335 17927 consensus_queue.cc:260] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee52c09beab44ec2a44279a09252a8fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } } peers { permanent_uuid: "cb3c446f02494636b9c4d80ab500cc3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "ec6c330979de44deaeaa07281758d659" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260619 12:39:24.513331 17928 consensus_queue.cc:260] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee52c09beab44ec2a44279a09252a8fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } } peers { permanent_uuid: "cb3c446f02494636b9c4d80ab500cc3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "ec6c330979de44deaeaa07281758d659" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260619 12:39:24.513815 17926 ts_tablet_manager.cc:1428] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260619 12:39:24.514045 17775 heartbeater.cc:503] Master 127.0.98.62:39647 was elected leader, sending a full tablet report...
I20260619 12:39:24.514135 17641 heartbeater.cc:503] Master 127.0.98.62:39647 was elected leader, sending a full tablet report...
I20260619 12:39:24.514305 17909 heartbeater.cc:503] Master 127.0.98.62:39647 was elected leader, sending a full tablet report...
I20260619 12:39:24.515344 17928 ts_tablet_manager.cc:1428] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260619 12:39:24.515805 17927 ts_tablet_manager.cc:1428] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 12:39:24.607309 17932 raft_consensus.cc:482] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:39:24.607589 17932 raft_consensus.cc:504] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee52c09beab44ec2a44279a09252a8fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } } peers { permanent_uuid: "cb3c446f02494636b9c4d80ab500cc3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "ec6c330979de44deaeaa07281758d659" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260619 12:39:24.608676 17932 leader_election.cc:288] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee52c09beab44ec2a44279a09252a8fe (127.0.98.3:39591), cb3c446f02494636b9c4d80ab500cc3b (127.0.98.2:45895)
I20260619 12:39:24.613042 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.609554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60785 (local address 127.0.98.2:45895)
0619 12:39:24.609718 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:24.609723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:24.609777 (+ 54us) server_negotiation.cc:407] Connection header received
0619 12:39:24.609913 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:24.609916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:24.609965 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:24.610046 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:24.610389 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.610886 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.612024 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.612463 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.612578 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.612644 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:24.612831 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 12:39:24.612907 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 12:39:24.613058 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.609180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52371 (local address 127.0.98.3:39591)
0619 12:39:24.609315 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:24.609319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:24.609370 (+ 51us) server_negotiation.cc:407] Connection header received
0619 12:39:24.609578 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:24.609581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:24.609643 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:24.609695 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:24.610194 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.610819 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.612064 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.612474 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.612607 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.612672 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:24.612880 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 12:39:24.612955 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:39:24.613337 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.609394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45895 (local address 127.0.98.1:60785)
0619 12:39:24.609700 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:24.609712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:24.609809 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:24.610087 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:24.610090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:24.610141 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:24.610270 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:24.610276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.611006 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.611009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:24.611901 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:24.611910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.612586 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.612589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:24.612689 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.612716 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 12:39:24.612785 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 12:39:24.612879 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":19,"thread_start_us":88,"threads_started":1}
I20260619 12:39:24.613075 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.608953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39591 (local address 127.0.98.1:52371)
0619 12:39:24.609286 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:24.609306 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:24.609418 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:24.609887 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:24.609890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:24.609909 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:24.610059 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:24.610067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.610938 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.610944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:24.611917 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:24.611926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.612619 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.612622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:24.612728 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.612748 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 12:39:24.612846 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 12:39:24.612912 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":23,"thread_start_us":134,"threads_started":1}
I20260619 12:39:24.614343 17729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21be0899a1af45f8895e54a997c4f6cc" candidate_uuid: "ec6c330979de44deaeaa07281758d659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3c446f02494636b9c4d80ab500cc3b" is_pre_election: true
I20260619 12:39:24.614342 17863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21be0899a1af45f8895e54a997c4f6cc" candidate_uuid: "ec6c330979de44deaeaa07281758d659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee52c09beab44ec2a44279a09252a8fe" is_pre_election: true
I20260619 12:39:24.614707 17863 raft_consensus.cc:2454] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec6c330979de44deaeaa07281758d659 in term 0.
I20260619 12:39:24.614707 17729 raft_consensus.cc:2454] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec6c330979de44deaeaa07281758d659 in term 0.
I20260619 12:39:24.615212 17529 leader_election.cc:302] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb3c446f02494636b9c4d80ab500cc3b, ec6c330979de44deaeaa07281758d659; no voters:
I20260619 12:39:24.615512 17932 raft_consensus.cc:2781] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:39:24.615657 17932 raft_consensus.cc:482] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:39:24.615738 17932 raft_consensus.cc:3037] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:24.616854 17932 raft_consensus.cc:504] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee52c09beab44ec2a44279a09252a8fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } } peers { permanent_uuid: "cb3c446f02494636b9c4d80ab500cc3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "ec6c330979de44deaeaa07281758d659" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260619 12:39:24.617231 17932 leader_election.cc:288] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [CANDIDATE]: Term 1 election: Requested vote from peers ee52c09beab44ec2a44279a09252a8fe (127.0.98.3:39591), cb3c446f02494636b9c4d80ab500cc3b (127.0.98.2:45895)
I20260619 12:39:24.617718 17729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21be0899a1af45f8895e54a997c4f6cc" candidate_uuid: "ec6c330979de44deaeaa07281758d659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3c446f02494636b9c4d80ab500cc3b"
I20260619 12:39:24.617718 17863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21be0899a1af45f8895e54a997c4f6cc" candidate_uuid: "ec6c330979de44deaeaa07281758d659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee52c09beab44ec2a44279a09252a8fe"
I20260619 12:39:24.617865 17863 raft_consensus.cc:3037] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:24.617865 17729 raft_consensus.cc:3037] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:24.619300 17729 raft_consensus.cc:2454] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6c330979de44deaeaa07281758d659 in term 1.
I20260619 12:39:24.619299 17863 raft_consensus.cc:2454] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6c330979de44deaeaa07281758d659 in term 1.
I20260619 12:39:24.619683 17529 leader_election.cc:302] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb3c446f02494636b9c4d80ab500cc3b, ec6c330979de44deaeaa07281758d659; no voters:
I20260619 12:39:24.619884 17932 raft_consensus.cc:2781] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:24.620121 17932 raft_consensus.cc:686] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [term 1 LEADER]: Becoming Leader. State: Replica: ec6c330979de44deaeaa07281758d659, State: Running, Role: LEADER
I20260619 12:39:24.620503 17932 consensus_queue.cc:237] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee52c09beab44ec2a44279a09252a8fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } } peers { permanent_uuid: "cb3c446f02494636b9c4d80ab500cc3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "ec6c330979de44deaeaa07281758d659" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260619 12:39:24.624258 17446 catalog_manager.cc:5495] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec6c330979de44deaeaa07281758d659 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ec6c330979de44deaeaa07281758d659" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee52c09beab44ec2a44279a09252a8fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb3c446f02494636b9c4d80ab500cc3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6c330979de44deaeaa07281758d659" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } health_report { overall_health: HEALTHY } } }
I20260619 12:39:24.644645 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.640542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.0.98.1:33255)
0619 12:39:24.640689 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:24.640695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:24.640716 (+ 21us) server_negotiation.cc:407] Connection header received
0619 12:39:24.640819 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:24.640822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:24.640878 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:24.640996 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:39:24.641255 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.641782 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.642781 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.643010 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.643173 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.643585 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:39:24.644065 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:39:24.644175 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:24.644333 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 12:39:24.644394 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 12:39:24.650995 17863 raft_consensus.cc:1261] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe [term 1 FOLLOWER]: Refusing update from remote peer ec6c330979de44deaeaa07281758d659: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:39:24.650995 17729 raft_consensus.cc:1261] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b [term 1 FOLLOWER]: Refusing update from remote peer ec6c330979de44deaeaa07281758d659: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:39:24.651861 17932 consensus_queue.cc:1035] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3c446f02494636b9c4d80ab500cc3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:39:24.652104 17937 consensus_queue.cc:1035] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee52c09beab44ec2a44279a09252a8fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:39:24.663360 17941 mvcc.cc:205] Tried to move back new op lower bound from 7298550844002271232 to 7298550843888803840. Current Snapshot: MvccSnapshot[applied={T|T < 7298550844002271232}]
I20260619 12:39:24.670912 17446 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:39:24.671123 17446 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:39:24.673868 17446 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab75f7a2a4154e81adacad74471dbf5d: Sending DeleteTablet for 3 replicas of tablet 21be0899a1af45f8895e54a997c4f6cc
I20260619 12:39:24.674835 17575 tablet_service.cc:1514] Processing DeleteTablet for tablet 21be0899a1af45f8895e54a997c4f6cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52842
I20260619 12:39:24.674846 17843 tablet_service.cc:1514] Processing DeleteTablet for tablet 21be0899a1af45f8895e54a997c4f6cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39700
I20260619 12:39:24.674958 17709 tablet_service.cc:1514] Processing DeleteTablet for tablet 21be0899a1af45f8895e54a997c4f6cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39402
I20260619 12:39:24.675405 17950 tablet_replica.cc:330] T 21be0899a1af45f8895e54a997c4f6cc P ec6c330979de44deaeaa07281758d659: stopping tablet replica
I20260619 12:39:24.675524 17952 tablet_replica.cc:330] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe: stopping tablet replica
I20260619 12:39:24.675630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 17512
I20260619 12:39:24.675633 17951 tablet_replica.cc:330] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b: stopping tablet replica
I20260619 12:39:24.675987 17951 raft_consensus.cc:2229] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:39:24.676446 17951 raft_consensus.cc:2258] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:39:24.676460 17952 raft_consensus.cc:2229] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:39:24.676759 17952 raft_consensus.cc:2258] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:39:24.678987 17951 ts_tablet_manager.cc:1905] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:39:24.680368 17952 ts_tablet_manager.cc:1905] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:39:24.681996 17951 ts_tablet_manager.cc:1918] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:39:24.682116 17951 log.cc:1196] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/21be0899a1af45f8895e54a997c4f6cc
I20260619 12:39:24.682446 17951 ts_tablet_manager.cc:1939] T 21be0899a1af45f8895e54a997c4f6cc P cb3c446f02494636b9c4d80ab500cc3b: Deleting consensus metadata
I20260619 12:39:24.683177 17952 ts_tablet_manager.cc:1918] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:39:24.683266 17952 log.cc:1196] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/21be0899a1af45f8895e54a997c4f6cc
I20260619 12:39:24.683622 17952 ts_tablet_manager.cc:1939] T 21be0899a1af45f8895e54a997c4f6cc P ee52c09beab44ec2a44279a09252a8fe: Deleting consensus metadata
W20260619 12:39:24.683831 17435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33255 (error 108)
I20260619 12:39:24.684358 17435 catalog_manager.cc:4841] TS cb3c446f02494636b9c4d80ab500cc3b (127.0.98.2:45895): tablet 21be0899a1af45f8895e54a997c4f6cc (table test-table [id=3982041207ac4a2d8eb5ea2f4cdaf43d]) successfully deleted
I20260619 12:39:24.684914 17435 catalog_manager.cc:4841] TS ee52c09beab44ec2a44279a09252a8fe (127.0.98.3:39591): tablet 21be0899a1af45f8895e54a997c4f6cc (table test-table [id=3982041207ac4a2d8eb5ea2f4cdaf43d]) successfully deleted
W20260619 12:39:24.685045 17924 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:39:24.684683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33255 (local address 127.0.0.1:52860)
0619 12:39:24.684841 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:24.684912 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33255: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 12:39:24.685201 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 17646
W20260619 12:39:24.688370 17435 catalog_manager.cc:4584] TS ec6c330979de44deaeaa07281758d659 (127.0.98.1:33255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21be0899a1af45f8895e54a997c4f6cc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33255: connect: Connection reset by peer (error 104)
I20260619 12:39:24.692298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 17780
I20260619 12:39:24.698222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 17418
2026-06-19T12:39:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2388 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T12:39:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:24Z Disabled control of system clock
I20260619 12:39:24.729120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46733
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:39:24.842067 17957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:24.842386 17957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:24.842422 17957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:24.846576 17957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:24.846652 17957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:24.846666 17957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:24.846689 17957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:24.846704 17957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:24.851670 17957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:24.852849 17957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:24.854180 17957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:24.862838 17963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:24.862840 17962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:24.862929 17957 server_base.cc:1034] running on GCE node
W20260619 12:39:24.862840 17965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:24.863768 17957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:24.864938 17957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:24.866130 17957 hybrid_clock.cc:648] HybridClock initialized: now 1781872764866096 us; error 46 us; skew 500 ppm
I20260619 12:39:24.868290 17957 webserver.cc:460] Webserver started at http://127.0.98.62:41731/ using document root <none> and password file <none>
I20260619 12:39:24.868928 17957 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:24.869000 17957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:24.869236 17957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:24.871426 17957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "6361c2aef77846418478e2fc3b9e5af9"
format_stamp: "Formatted at 2026-06-19 12:39:24 on dist-test-slave-kdrb"
server_key: "6e27903c54f3831a6b8b3524ac0777f7"
server_key_iv: "459ba35774a0167f56077e9452961bdf"
server_key_version: "encryptionkey@0"
I20260619 12:39:24.872124 17957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "6361c2aef77846418478e2fc3b9e5af9"
format_stamp: "Formatted at 2026-06-19 12:39:24 on dist-test-slave-kdrb"
server_key: "6e27903c54f3831a6b8b3524ac0777f7"
server_key_iv: "459ba35774a0167f56077e9452961bdf"
server_key_version: "encryptionkey@0"
I20260619 12:39:24.876077 17957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 12:39:24.878679 17971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:24.880276 17957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:24.880434 17957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "6361c2aef77846418478e2fc3b9e5af9"
format_stamp: "Formatted at 2026-06-19 12:39:24 on dist-test-slave-kdrb"
server_key: "6e27903c54f3831a6b8b3524ac0777f7"
server_key_iv: "459ba35774a0167f56077e9452961bdf"
server_key_version: "encryptionkey@0"
I20260619 12:39:24.880622 17957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:24.896920 17957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:24.897675 17957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:24.897902 17957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:24.905484 17957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46733
I20260619 12:39:24.905501 18023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46733 every 8 connection(s)
I20260619 12:39:24.906522 17957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:24.909789 18024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:24.915148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 17957
I20260619 12:39:24.915292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:24.915634 392 external_mini_cluster.cc:1428] Setting key 440dba167ed9a93041a11f0e862d5ddd
I20260619 12:39:24.916115 18024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9: Bootstrap starting.
I20260619 12:39:24.918838 18024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:24.919720 18024 log.cc:824] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:24.921751 18024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9: No bootstrap required, opened a new log
I20260619 12:39:24.925560 18024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6361c2aef77846418478e2fc3b9e5af9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } }
I20260619 12:39:24.925731 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:24.917255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43096 (local address 127.0.98.62:46733)
0619 12:39:24.918035 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:24.918050 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:24.918089 (+ 39us) server_negotiation.cc:407] Connection header received
0619 12:39:24.918776 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:24.918795 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:24.919315 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:24.919769 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:24.920267 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.921284 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.922264 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:24.922791 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:24.922974 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:24.923927 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:24.923932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:24.923947 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:24.924361 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:24.924419 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:24.924440 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:24.924555 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:24.924676 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 12:39:24.924954 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":173,"threads_started":1}
I20260619 12:39:24.925835 18024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:24.925921 18024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6361c2aef77846418478e2fc3b9e5af9, State: Initialized, Role: FOLLOWER
I20260619 12:39:24.926438 18024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6361c2aef77846418478e2fc3b9e5af9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } }
I20260619 12:39:24.926602 18024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:24.926657 18024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:24.926752 18024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:24.927819 18024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6361c2aef77846418478e2fc3b9e5af9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } }
I20260619 12:39:24.928226 18024 leader_election.cc:302] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6361c2aef77846418478e2fc3b9e5af9; no voters:
I20260619 12:39:24.928578 18024 leader_election.cc:288] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:24.928772 18029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:24.929042 18029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [term 1 LEADER]: Becoming Leader. State: Replica: 6361c2aef77846418478e2fc3b9e5af9, State: Running, Role: LEADER
I20260619 12:39:24.929368 18029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6361c2aef77846418478e2fc3b9e5af9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } }
I20260619 12:39:24.929965 18024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:24.931207 18030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6361c2aef77846418478e2fc3b9e5af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6361c2aef77846418478e2fc3b9e5af9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } } }
I20260619 12:39:24.931226 18031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6361c2aef77846418478e2fc3b9e5af9. Latest consensus state: current_term: 1 leader_uuid: "6361c2aef77846418478e2fc3b9e5af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6361c2aef77846418478e2fc3b9e5af9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } } }
I20260619 12:39:24.931363 18031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:24.931363 18030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9 [sys.catalog]: This master's current role is: LEADER
W20260619 12:39:24.935088 18043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:39:24.935204 18043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:39:24.935294 18044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:24.936214 18044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:24.942106 18044 catalog_manager.cc:1347] Generated new cluster ID: c671407e85dd4499a0e383265a4a4995
I20260619 12:39:24.942193 18044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:24.953991 18044 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:24.954979 18044 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:24.960279 18044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9: Generated new TSK 0
I20260619 12:39:24.961072 18044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:24.964699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--builtin_ntp_servers=127.0.98.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:39:25.084429 18048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:25.084789 18048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:25.084836 18048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:25.089043 18048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:25.089115 18048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:25.089218 18048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:39:25.095664 18048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:25.096889 18048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:25.098328 18048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:25.108227 18054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:25.108270 18053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:25.108388 18048 server_base.cc:1034] running on GCE node
W20260619 12:39:25.108577 18056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:25.109033 18048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:25.109735 18048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:25.110941 18048 hybrid_clock.cc:648] HybridClock initialized: now 1781872765110922 us; error 34 us; skew 500 ppm
I20260619 12:39:25.113015 18048 webserver.cc:460] Webserver started at http://127.0.98.1:42289/ using document root <none> and password file <none>
I20260619 12:39:25.113662 18048 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:25.113757 18048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:25.114001 18048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:25.116307 18048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "c8eac9982e874252a0ccb8ba94682715"
format_stamp: "Formatted at 2026-06-19 12:39:25 on dist-test-slave-kdrb"
server_key: "2447069cca776b8ba0cb9f7cb56e5777"
server_key_iv: "e4319917f8d8deff3ab99ddfda15ad18"
server_key_version: "encryptionkey@0"
I20260619 12:39:25.116907 18048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c8eac9982e874252a0ccb8ba94682715"
format_stamp: "Formatted at 2026-06-19 12:39:25 on dist-test-slave-kdrb"
server_key: "2447069cca776b8ba0cb9f7cb56e5777"
server_key_iv: "e4319917f8d8deff3ab99ddfda15ad18"
server_key_version: "encryptionkey@0"
I20260619 12:39:25.120905 18048 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:39:25.123730 18062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:25.125057 18048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:25.125207 18048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "c8eac9982e874252a0ccb8ba94682715"
format_stamp: "Formatted at 2026-06-19 12:39:25 on dist-test-slave-kdrb"
server_key: "2447069cca776b8ba0cb9f7cb56e5777"
server_key_iv: "e4319917f8d8deff3ab99ddfda15ad18"
server_key_version: "encryptionkey@0"
I20260619 12:39:25.125346 18048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:25.139024 18048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:25.139909 18048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:25.140121 18048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:25.140755 18048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:25.141736 18048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:25.141811 18048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:25.141882 18048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:25.141925 18048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:25.151193 18048 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33147
I20260619 12:39:25.151216 18175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33147 every 8 connection(s)
I20260619 12:39:25.152182 18048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:39:25.161859 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.154058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54413 (local address 127.0.98.62:46733)
0619 12:39:25.154208 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:25.154212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:25.155371 (+ 1159us) server_negotiation.cc:407] Connection header received
0619 12:39:25.156840 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:25.156843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:25.156906 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:25.157015 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:25.157950 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.158502 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.159768 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.159991 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.160106 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.161333 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:25.161336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:25.161338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:25.161352 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:25.161373 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:25.161380 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:25.161463 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:25.161631 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 12:39:25.161702 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 12:39:25.162328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 18048
I20260619 12:39:25.162469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:39:25.162782 392 external_mini_cluster.cc:1428] Setting key 0e6d2cb6e05d41a18ae1b5569f447d5d
I20260619 12:39:25.162990 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.154505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46733 (local address 127.0.98.1:54413)
0619 12:39:25.155201 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:25.155245 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:25.156603 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:25.157202 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:25.157209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:25.157299 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:25.157807 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:25.157820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.158644 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.158647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:25.159633 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:25.159642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.160149 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.160153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:25.160338 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.161014 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:39:25.161015 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:25.161060 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:39:25.161131 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:25.161464 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:25.161476 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:25.161484 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 12:39:25.161630 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 12:39:25.161989 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":120,"threads_started":1}
I20260619 12:39:25.164412 18176 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46733
I20260619 12:39:25.164680 18176 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:25.165030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--builtin_ntp_servers=127.0.98.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:39:25.165350 18176 heartbeater.cc:511] Master 127.0.98.62:46733 requested a full tablet report, sending...
I20260619 12:39:25.167215 17988 ts_manager.cc:194] Registered new tserver with Master: c8eac9982e874252a0ccb8ba94682715 (127.0.98.1:33147)
I20260619 12:39:25.168563 17988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:54413
W20260619 12:39:25.279552 18179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:25.279888 18179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:25.279966 18179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:25.284294 18179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:25.284384 18179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:25.284507 18179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:39:25.289624 18179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:25.290851 18179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:25.292435 18179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:25.302587 18185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:25.302591 18184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:25.302591 18187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:25.302798 18179 server_base.cc:1034] running on GCE node
I20260619 12:39:25.303385 18179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:25.304173 18179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:25.305383 18179 hybrid_clock.cc:648] HybridClock initialized: now 1781872765305367 us; error 30 us; skew 500 ppm
I20260619 12:39:25.307453 18179 webserver.cc:460] Webserver started at http://127.0.98.2:37355/ using document root <none> and password file <none>
I20260619 12:39:25.308145 18179 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:25.308213 18179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:25.308485 18179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:25.310710 18179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "3efde9dc9a4749c28c19e4d7b5158bcf"
format_stamp: "Formatted at 2026-06-19 12:39:25 on dist-test-slave-kdrb"
server_key: "b82736d27694611e8d1674259bb16297"
server_key_iv: "08d514b2290dd88569c59af22ffb1ac9"
server_key_version: "encryptionkey@0"
I20260619 12:39:25.311340 18179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3efde9dc9a4749c28c19e4d7b5158bcf"
format_stamp: "Formatted at 2026-06-19 12:39:25 on dist-test-slave-kdrb"
server_key: "b82736d27694611e8d1674259bb16297"
server_key_iv: "08d514b2290dd88569c59af22ffb1ac9"
server_key_version: "encryptionkey@0"
I20260619 12:39:25.315215 18179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 12:39:25.317814 18193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:25.319077 18179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:39:25.319229 18179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "3efde9dc9a4749c28c19e4d7b5158bcf"
format_stamp: "Formatted at 2026-06-19 12:39:25 on dist-test-slave-kdrb"
server_key: "b82736d27694611e8d1674259bb16297"
server_key_iv: "08d514b2290dd88569c59af22ffb1ac9"
server_key_version: "encryptionkey@0"
I20260619 12:39:25.319365 18179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:25.344274 18179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:25.345070 18179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:25.345295 18179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:25.345937 18179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:25.346918 18179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:25.346989 18179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:25.347065 18179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:25.347106 18179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:25.356068 18179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43523
I20260619 12:39:25.356088 18306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43523 every 8 connection(s)
I20260619 12:39:25.357025 18179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:39:25.362792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 18179
I20260619 12:39:25.362931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:39:25.363265 392 external_mini_cluster.cc:1428] Setting key 920d1cf85cbe4b34a73c5e0fb19b48bd
I20260619 12:39:25.365458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--builtin_ntp_servers=127.0.98.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:39:25.366941 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.358855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50345 (local address 127.0.98.62:46733)
0619 12:39:25.358995 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:25.358999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:25.360142 (+ 1143us) server_negotiation.cc:407] Connection header received
0619 12:39:25.361513 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:25.361515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:25.361565 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:25.361650 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:25.362573 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.363408 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.364809 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.365095 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.365206 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.366400 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:25.366406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:25.366413 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:25.366431 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:25.366459 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:25.366473 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:25.366556 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:25.366732 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 12:39:25.366798 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:39:25.367937 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.359282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46733 (local address 127.0.98.2:50345)
0619 12:39:25.359988 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:25.360029 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:25.361298 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:25.361829 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:25.361835 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:25.361920 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:25.362406 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:25.362417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.363568 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.363571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:25.364677 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:25.364690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.365246 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.365249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:25.365399 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.366123 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:39:25.366125 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:25.366169 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:39:25.366237 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:25.366561 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:25.366568 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:25.366572 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:39:25.366702 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 12:39:25.367036 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":174,"threads_started":1}
I20260619 12:39:25.369194 18307 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46733
I20260619 12:39:25.369452 18307 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:25.369990 18307 heartbeater.cc:511] Master 127.0.98.62:46733 requested a full tablet report, sending...
I20260619 12:39:25.371147 17988 ts_manager.cc:194] Registered new tserver with Master: 3efde9dc9a4749c28c19e4d7b5158bcf (127.0.98.2:43523)
I20260619 12:39:25.372092 17988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50345
W20260619 12:39:25.482187 18310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:25.482473 18310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:25.482513 18310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:25.486819 18310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:25.486897 18310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:25.486955 18310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:39:25.492063 18310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:25.493194 18310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:25.494594 18310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:25.505239 18315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:25.505280 18310 server_base.cc:1034] running on GCE node
W20260619 12:39:25.505252 18318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:25.505252 18316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:25.506038 18310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:25.506755 18310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:25.507944 18310 hybrid_clock.cc:648] HybridClock initialized: now 1781872765507928 us; error 32 us; skew 500 ppm
I20260619 12:39:25.509837 18310 webserver.cc:460] Webserver started at http://127.0.98.3:36377/ using document root <none> and password file <none>
I20260619 12:39:25.510468 18310 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:25.510567 18310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:25.510811 18310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:25.513159 18310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "9fcbdad1030d442ebd1f996e3b9c154b"
format_stamp: "Formatted at 2026-06-19 12:39:25 on dist-test-slave-kdrb"
server_key: "b9f761a71c70ba407a97697206faad8d"
server_key_iv: "b8d9c031c11abe780a6d009c9d9d54d0"
server_key_version: "encryptionkey@0"
I20260619 12:39:25.513772 18310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9fcbdad1030d442ebd1f996e3b9c154b"
format_stamp: "Formatted at 2026-06-19 12:39:25 on dist-test-slave-kdrb"
server_key: "b9f761a71c70ba407a97697206faad8d"
server_key_iv: "b8d9c031c11abe780a6d009c9d9d54d0"
server_key_version: "encryptionkey@0"
I20260619 12:39:25.517650 18310 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:39:25.520283 18324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:25.521559 18310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 12:39:25.521709 18310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "9fcbdad1030d442ebd1f996e3b9c154b"
format_stamp: "Formatted at 2026-06-19 12:39:25 on dist-test-slave-kdrb"
server_key: "b9f761a71c70ba407a97697206faad8d"
server_key_iv: "b8d9c031c11abe780a6d009c9d9d54d0"
server_key_version: "encryptionkey@0"
I20260619 12:39:25.521839 18310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:25.542088 18310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:25.542994 18310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:25.543205 18310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:25.543903 18310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:25.544886 18310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:25.544958 18310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:25.545030 18310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:25.545061 18310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:25.554100 18310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41413
I20260619 12:39:25.554180 18437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41413 every 8 connection(s)
I20260619 12:39:25.555114 18310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:39:25.562587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 18310
I20260619 12:39:25.562719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:39:25.563030 392 external_mini_cluster.cc:1428] Setting key 93dd4b8d365a906a50bd43582cd087a7
I20260619 12:39:25.564524 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.557031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38663 (local address 127.0.98.62:46733)
0619 12:39:25.557178 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:25.557182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:25.558292 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 12:39:25.559625 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:25.559627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:25.559676 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:25.559754 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:25.560597 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.561120 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.562280 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.562624 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.562744 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.564003 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:25.564007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:25.564010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:25.564023 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:25.564050 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:25.564058 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:25.564135 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:25.564319 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 12:39:25.564383 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:39:25.565557 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.557478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46733 (local address 127.0.98.3:38663)
0619 12:39:25.558115 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:25.558168 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:25.559414 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:25.559886 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:25.559893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:25.559978 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:25.560476 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:25.560488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.561231 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.561234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:25.562138 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:25.562174 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.562823 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.562827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:25.562986 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.563719 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:39:25.563721 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:25.563765 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:39:25.563835 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:25.564148 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:25.564159 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:25.564167 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 12:39:25.564318 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 12:39:25.564671 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":110,"threads_started":1}
I20260619 12:39:25.566681 18438 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46733
I20260619 12:39:25.566902 18438 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:25.567414 18438 heartbeater.cc:511] Master 127.0.98.62:46733 requested a full tablet report, sending...
I20260619 12:39:25.568477 17988 ts_manager.cc:194] Registered new tserver with Master: 9fcbdad1030d442ebd1f996e3b9c154b (127.0.98.3:41413)
I20260619 12:39:25.569056 17988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38663
I20260619 12:39:25.577384 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:39:25.582882 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.579336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43100 (local address 127.0.98.62:46733)
0619 12:39:25.579466 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:25.579508 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:25.579650 (+ 142us) server_negotiation.cc:407] Connection header received
0619 12:39:25.579753 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:25.579755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:25.579796 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:25.579866 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:25.580111 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.580619 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.581435 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.581637 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.581711 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.582521 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:25.582524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:25.582526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:25.582538 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:25.582555 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:25.582563 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:25.582620 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:25.582712 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 12:39:25.582764 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:39:25.585794 17988 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:39:25.588853 17988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:39:25.603945 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.597517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.0.98.2:43523)
0619 12:39:25.598303 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:25.598315 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:25.598341 (+ 26us) server_negotiation.cc:407] Connection header received
0619 12:39:25.598423 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:25.598431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:25.598685 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:25.598874 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:25.599205 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.600381 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.602552 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.603127 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.603267 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.603468 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:25.603642 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 12:39:25.603756 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":209,"threads_started":1}
I20260619 12:39:25.603941 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.597341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43523 (local address 127.0.0.1:54654)
0619 12:39:25.597786 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:25.597841 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:25.597975 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:25.598866 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:25.598872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:25.598895 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:25.599045 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:25.599053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.600496 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.600503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:25.602420 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:25.602430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.603279 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.603283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:25.603398 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.603441 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 12:39:25.603609 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 12:39:25.603756 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"spinlock_wait_cycles":4864,"thread_start_us":121,"threads_started":1}
I20260619 12:39:25.604288 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.597545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.0.98.1:33147)
0619 12:39:25.599201 (+ 1656us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:25.599213 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:25.599242 (+ 29us) server_negotiation.cc:407] Connection header received
0619 12:39:25.599306 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:25.599309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:25.599454 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:25.599659 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:25.599886 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.600952 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.602439 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.603158 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.603265 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.603505 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:25.604007 (+ 502us) server_negotiation.cc:299] Negotiation successful
0619 12:39:25.604109 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1537,"thread_start_us":142,"threads_started":1}
I20260619 12:39:25.604538 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.597945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33147 (local address 127.0.0.1:55050)
0619 12:39:25.598830 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:25.598843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:25.598925 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:25.599635 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:25.599638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:25.599657 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:25.599760 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:25.599770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.601084 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.601089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:25.602304 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:25.602316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.603555 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.603559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:25.603694 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.603751 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 12:39:25.603815 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 12:39:25.603914 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":86,"thread_start_us":96,"threads_started":1}
I20260619 12:39:25.604867 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.598186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41413 (local address 127.0.0.1:60916)
0619 12:39:25.599060 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:25.599073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:25.599185 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:25.599949 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:25.599952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:25.599971 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:25.600074 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:25.600081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.601415 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.601420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:25.602195 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:25.602204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.603555 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.603559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:25.603698 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.603740 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 12:39:25.603820 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 12:39:25.604591 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":713,"thread_start_us":100,"threads_started":1}
I20260619 12:39:25.605188 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.598358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60916 (local address 127.0.98.3:41413)
0619 12:39:25.598701 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:25.598717 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:25.599306 (+ 589us) server_negotiation.cc:407] Connection header received
0619 12:39:25.599397 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:25.599402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:25.599639 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:25.599793 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:25.600205 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.601290 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.602325 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.603158 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.603265 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.603489 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:25.603914 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 12:39:25.605022 (+ 1108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":92,"threads_started":1}
I20260619 12:39:25.607547 18110 tablet_service.cc:1467] Processing CreateTablet for tablet ab06aa00abca478999b3cf9ab402593e (DEFAULT_TABLE table=test-table [id=5b096b624234473c800dc89d68af58d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:25.607545 18241 tablet_service.cc:1467] Processing CreateTablet for tablet ab06aa00abca478999b3cf9ab402593e (DEFAULT_TABLE table=test-table [id=5b096b624234473c800dc89d68af58d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:25.607854 18372 tablet_service.cc:1467] Processing CreateTablet for tablet ab06aa00abca478999b3cf9ab402593e (DEFAULT_TABLE table=test-table [id=5b096b624234473c800dc89d68af58d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:25.608739 18372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab06aa00abca478999b3cf9ab402593e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:25.608747 18241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab06aa00abca478999b3cf9ab402593e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:25.608747 18110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab06aa00abca478999b3cf9ab402593e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:25.615746 18455 tablet_bootstrap.cc:492] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b: Bootstrap starting.
I20260619 12:39:25.616564 18456 tablet_bootstrap.cc:492] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf: Bootstrap starting.
I20260619 12:39:25.616995 18457 tablet_bootstrap.cc:492] T ab06aa00abca478999b3cf9ab402593e P c8eac9982e874252a0ccb8ba94682715: Bootstrap starting.
I20260619 12:39:25.618156 18455 tablet_bootstrap.cc:654] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:25.618999 18456 tablet_bootstrap.cc:654] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:25.619055 18455 log.cc:824] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:25.619091 18457 tablet_bootstrap.cc:654] T ab06aa00abca478999b3cf9ab402593e P c8eac9982e874252a0ccb8ba94682715: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:25.619992 18456 log.cc:824] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:25.620096 18457 log.cc:824] T ab06aa00abca478999b3cf9ab402593e P c8eac9982e874252a0ccb8ba94682715: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:25.621387 18455 tablet_bootstrap.cc:492] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b: No bootstrap required, opened a new log
I20260619 12:39:25.621593 18455 ts_tablet_manager.cc:1397] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:39:25.622082 18457 tablet_bootstrap.cc:492] T ab06aa00abca478999b3cf9ab402593e P c8eac9982e874252a0ccb8ba94682715: No bootstrap required, opened a new log
I20260619 12:39:25.622262 18456 tablet_bootstrap.cc:492] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf: No bootstrap required, opened a new log
I20260619 12:39:25.622296 18457 ts_tablet_manager.cc:1397] T ab06aa00abca478999b3cf9ab402593e P c8eac9982e874252a0ccb8ba94682715: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:39:25.622453 18456 ts_tablet_manager.cc:1397] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:39:25.626091 18457 raft_consensus.cc:348] T ab06aa00abca478999b3cf9ab402593e P c8eac9982e874252a0ccb8ba94682715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efde9dc9a4749c28c19e4d7b5158bcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "c8eac9982e874252a0ccb8ba94682715" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33147 } } peers { permanent_uuid: "9fcbdad1030d442ebd1f996e3b9c154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41413 } }
I20260619 12:39:25.626396 18457 raft_consensus.cc:374] T ab06aa00abca478999b3cf9ab402593e P c8eac9982e874252a0ccb8ba94682715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:25.626238 18455 raft_consensus.cc:348] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efde9dc9a4749c28c19e4d7b5158bcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "c8eac9982e874252a0ccb8ba94682715" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33147 } } peers { permanent_uuid: "9fcbdad1030d442ebd1f996e3b9c154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41413 } }
I20260619 12:39:25.626458 18457 raft_consensus.cc:729] T ab06aa00abca478999b3cf9ab402593e P c8eac9982e874252a0ccb8ba94682715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8eac9982e874252a0ccb8ba94682715, State: Initialized, Role: FOLLOWER
I20260619 12:39:25.626507 18455 raft_consensus.cc:374] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:25.626570 18455 raft_consensus.cc:729] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fcbdad1030d442ebd1f996e3b9c154b, State: Initialized, Role: FOLLOWER
I20260619 12:39:25.626780 18456 raft_consensus.cc:348] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efde9dc9a4749c28c19e4d7b5158bcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "c8eac9982e874252a0ccb8ba94682715" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33147 } } peers { permanent_uuid: "9fcbdad1030d442ebd1f996e3b9c154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41413 } }
I20260619 12:39:25.627015 18456 raft_consensus.cc:374] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:25.627063 18456 raft_consensus.cc:729] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efde9dc9a4749c28c19e4d7b5158bcf, State: Initialized, Role: FOLLOWER
I20260619 12:39:25.627072 18455 consensus_queue.cc:260] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efde9dc9a4749c28c19e4d7b5158bcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "c8eac9982e874252a0ccb8ba94682715" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33147 } } peers { permanent_uuid: "9fcbdad1030d442ebd1f996e3b9c154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41413 } }
I20260619 12:39:25.627053 18457 consensus_queue.cc:260] T ab06aa00abca478999b3cf9ab402593e P c8eac9982e874252a0ccb8ba94682715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efde9dc9a4749c28c19e4d7b5158bcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "c8eac9982e874252a0ccb8ba94682715" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33147 } } peers { permanent_uuid: "9fcbdad1030d442ebd1f996e3b9c154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41413 } }
I20260619 12:39:25.627460 18456 consensus_queue.cc:260] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efde9dc9a4749c28c19e4d7b5158bcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "c8eac9982e874252a0ccb8ba94682715" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33147 } } peers { permanent_uuid: "9fcbdad1030d442ebd1f996e3b9c154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41413 } }
I20260619 12:39:25.628190 18455 ts_tablet_manager.cc:1428] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 12:39:25.628217 18457 ts_tablet_manager.cc:1428] T ab06aa00abca478999b3cf9ab402593e P c8eac9982e874252a0ccb8ba94682715: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 12:39:25.628293 18456 ts_tablet_manager.cc:1428] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 12:39:25.628568 18438 heartbeater.cc:503] Master 127.0.98.62:46733 was elected leader, sending a full tablet report...
I20260619 12:39:25.628650 18176 heartbeater.cc:503] Master 127.0.98.62:46733 was elected leader, sending a full tablet report...
I20260619 12:39:25.629657 18307 heartbeater.cc:503] Master 127.0.98.62:46733 was elected leader, sending a full tablet report...
W20260619 12:39:25.653469 18177 tablet.cc:2369] T ab06aa00abca478999b3cf9ab402593e P c8eac9982e874252a0ccb8ba94682715: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:39:25.806272 18439 tablet.cc:2369] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:39:25.861975 18308 tablet.cc:2369] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:39:25.863713 18463 raft_consensus.cc:482] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:39:25.863950 18463 raft_consensus.cc:504] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efde9dc9a4749c28c19e4d7b5158bcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "c8eac9982e874252a0ccb8ba94682715" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33147 } } peers { permanent_uuid: "9fcbdad1030d442ebd1f996e3b9c154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41413 } }
I20260619 12:39:25.865191 18463 leader_election.cc:288] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8eac9982e874252a0ccb8ba94682715 (127.0.98.1:33147), 9fcbdad1030d442ebd1f996e3b9c154b (127.0.98.3:41413)
I20260619 12:39:25.869225 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.865618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38473 (local address 127.0.98.3:41413)
0619 12:39:25.865754 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:25.865760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:25.865934 (+ 174us) server_negotiation.cc:407] Connection header received
0619 12:39:25.866056 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:25.866059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:25.866104 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:25.866188 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:25.866529 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.867072 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.868213 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.868616 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.868753 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.868814 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:25.869029 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 12:39:25.869107 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 12:39:25.869225 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.865385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33147 (local address 127.0.98.2:50139)
0619 12:39:25.865602 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:25.865617 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:25.865719 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:25.866043 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:25.866046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:25.866066 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:25.866219 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:25.866228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.867077 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.867080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:25.868055 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:25.868065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.868735 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.868739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:25.868840 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.868868 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 12:39:25.868936 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 12:39:25.869032 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":35}
I20260619 12:39:25.869215 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.865562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50139 (local address 127.0.98.1:33147)
0619 12:39:25.865732 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:25.865738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:25.865786 (+ 48us) server_negotiation.cc:407] Connection header received
0619 12:39:25.865847 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:25.865851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:25.865918 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:25.865996 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:25.866370 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.866953 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.868188 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.868605 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.868715 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.868789 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:25.868985 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 12:39:25.869065 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 12:39:25.869233 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.865519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41413 (local address 127.0.98.2:38473)
0619 12:39:25.865847 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:25.865861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:25.865951 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:25.866230 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:25.866232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:25.866249 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:25.866413 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:25.866420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.867182 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.867184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:25.868068 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:25.868077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.868787 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.868792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:25.868885 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.868902 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 12:39:25.868999 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 12:39:25.869064 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":40,"thread_start_us":85,"threads_started":1}
I20260619 12:39:25.869937 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab06aa00abca478999b3cf9ab402593e" candidate_uuid: "3efde9dc9a4749c28c19e4d7b5158bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fcbdad1030d442ebd1f996e3b9c154b" is_pre_election: true
I20260619 12:39:25.869997 18130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab06aa00abca478999b3cf9ab402593e" candidate_uuid: "3efde9dc9a4749c28c19e4d7b5158bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8eac9982e874252a0ccb8ba94682715" is_pre_election: true
I20260619 12:39:25.870281 18392 raft_consensus.cc:2454] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efde9dc9a4749c28c19e4d7b5158bcf in term 0.
I20260619 12:39:25.870308 18130 raft_consensus.cc:2454] T ab06aa00abca478999b3cf9ab402593e P c8eac9982e874252a0ccb8ba94682715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efde9dc9a4749c28c19e4d7b5158bcf in term 0.
I20260619 12:39:25.870790 18197 leader_election.cc:302] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efde9dc9a4749c28c19e4d7b5158bcf, 9fcbdad1030d442ebd1f996e3b9c154b; no voters:
I20260619 12:39:25.871076 18463 raft_consensus.cc:2781] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:39:25.871155 18463 raft_consensus.cc:482] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:39:25.871193 18463 raft_consensus.cc:3037] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:25.872336 18463 raft_consensus.cc:504] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efde9dc9a4749c28c19e4d7b5158bcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "c8eac9982e874252a0ccb8ba94682715" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33147 } } peers { permanent_uuid: "9fcbdad1030d442ebd1f996e3b9c154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41413 } }
I20260619 12:39:25.872751 18463 leader_election.cc:288] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [CANDIDATE]: Term 1 election: Requested vote from peers c8eac9982e874252a0ccb8ba94682715 (127.0.98.1:33147), 9fcbdad1030d442ebd1f996e3b9c154b (127.0.98.3:41413)
I20260619 12:39:25.873114 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab06aa00abca478999b3cf9ab402593e" candidate_uuid: "3efde9dc9a4749c28c19e4d7b5158bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fcbdad1030d442ebd1f996e3b9c154b"
I20260619 12:39:25.873129 18130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab06aa00abca478999b3cf9ab402593e" candidate_uuid: "3efde9dc9a4749c28c19e4d7b5158bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8eac9982e874252a0ccb8ba94682715"
I20260619 12:39:25.873268 18392 raft_consensus.cc:3037] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:25.873236 18130 raft_consensus.cc:3037] T ab06aa00abca478999b3cf9ab402593e P c8eac9982e874252a0ccb8ba94682715 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:25.874298 18392 raft_consensus.cc:2454] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efde9dc9a4749c28c19e4d7b5158bcf in term 1.
I20260619 12:39:25.874341 18130 raft_consensus.cc:2454] T ab06aa00abca478999b3cf9ab402593e P c8eac9982e874252a0ccb8ba94682715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efde9dc9a4749c28c19e4d7b5158bcf in term 1.
I20260619 12:39:25.874661 18197 leader_election.cc:302] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efde9dc9a4749c28c19e4d7b5158bcf, 9fcbdad1030d442ebd1f996e3b9c154b; no voters:
I20260619 12:39:25.874868 18463 raft_consensus.cc:2781] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:25.875100 18463 raft_consensus.cc:686] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [term 1 LEADER]: Becoming Leader. State: Replica: 3efde9dc9a4749c28c19e4d7b5158bcf, State: Running, Role: LEADER
I20260619 12:39:25.875442 18463 consensus_queue.cc:237] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efde9dc9a4749c28c19e4d7b5158bcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "c8eac9982e874252a0ccb8ba94682715" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33147 } } peers { permanent_uuid: "9fcbdad1030d442ebd1f996e3b9c154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41413 } }
I20260619 12:39:25.878901 17988 catalog_manager.cc:5495] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efde9dc9a4749c28c19e4d7b5158bcf (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3efde9dc9a4749c28c19e4d7b5158bcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efde9dc9a4749c28c19e4d7b5158bcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8eac9982e874252a0ccb8ba94682715" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fcbdad1030d442ebd1f996e3b9c154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41413 } health_report { overall_health: UNKNOWN } } }
I20260619 12:39:25.897969 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.894308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54670 (local address 127.0.98.2:43523)
0619 12:39:25.894446 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:25.894449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:25.894503 (+ 54us) server_negotiation.cc:407] Connection header received
0619 12:39:25.894661 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:25.894663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:25.894722 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:25.894824 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:39:25.895088 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.895669 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.896493 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:25.896708 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:25.896791 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:25.897116 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:39:25.897598 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:39:25.897705 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:25.897755 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 12:39:25.897810 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:39:25.904361 18392 raft_consensus.cc:1261] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b [term 1 FOLLOWER]: Refusing update from remote peer 3efde9dc9a4749c28c19e4d7b5158bcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:39:25.904392 18130 raft_consensus.cc:1261] T ab06aa00abca478999b3cf9ab402593e P c8eac9982e874252a0ccb8ba94682715 [term 1 FOLLOWER]: Refusing update from remote peer 3efde9dc9a4749c28c19e4d7b5158bcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:39:25.905161 18465 consensus_queue.cc:1035] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8eac9982e874252a0ccb8ba94682715" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:39:25.905613 18463 consensus_queue.cc:1035] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fcbdad1030d442ebd1f996e3b9c154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:39:25.917066 18470 mvcc.cc:205] Tried to move back new op lower bound from 7298550849136033792 to 7298550849028534272. Current Snapshot: MvccSnapshot[applied={T|T < 7298550849136033792}]
I20260619 12:39:25.927176 17988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:39:25.927371 17988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:39:25.929728 17988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6361c2aef77846418478e2fc3b9e5af9: Sending DeleteTablet for 3 replicas of tablet ab06aa00abca478999b3cf9ab402593e
I20260619 12:39:25.930624 18241 tablet_service.cc:1514] Processing DeleteTablet for tablet ab06aa00abca478999b3cf9ab402593e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:25 UTC) from {username='slave'} at 127.0.0.1:54654
I20260619 12:39:25.930641 18372 tablet_service.cc:1514] Processing DeleteTablet for tablet ab06aa00abca478999b3cf9ab402593e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:25 UTC) from {username='slave'} at 127.0.0.1:60916
I20260619 12:39:25.931210 18478 tablet_replica.cc:330] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b: stopping tablet replica
I20260619 12:39:25.931207 18110 tablet_service.cc:1514] Processing DeleteTablet for tablet ab06aa00abca478999b3cf9ab402593e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:25 UTC) from {username='slave'} at 127.0.0.1:55050
I20260619 12:39:25.931723 18479 tablet_replica.cc:330] T ab06aa00abca478999b3cf9ab402593e P c8eac9982e874252a0ccb8ba94682715: stopping tablet replica
I20260619 12:39:25.931860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 18048
I20260619 12:39:25.931860 18477 tablet_replica.cc:330] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf: stopping tablet replica
I20260619 12:39:25.932307 18477 raft_consensus.cc:2229] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:39:25.932857 18477 raft_consensus.cc:2258] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:39:25.931579 18478 raft_consensus.cc:2229] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:39:25.934281 18478 raft_consensus.cc:2258] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:39:25.934507 18477 ts_tablet_manager.cc:1905] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:39:25.936065 18478 ts_tablet_manager.cc:1905] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:39:25.937652 18477 ts_tablet_manager.cc:1918] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:39:25.937767 18477 log.cc:1196] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/ab06aa00abca478999b3cf9ab402593e
I20260619 12:39:25.938089 18477 ts_tablet_manager.cc:1939] T ab06aa00abca478999b3cf9ab402593e P 3efde9dc9a4749c28c19e4d7b5158bcf: Deleting consensus metadata
I20260619 12:39:25.939790 18478 ts_tablet_manager.cc:1918] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:39:25.939805 17975 catalog_manager.cc:4841] TS 3efde9dc9a4749c28c19e4d7b5158bcf (127.0.98.2:43523): tablet ab06aa00abca478999b3cf9ab402593e (table test-table [id=5b096b624234473c800dc89d68af58d2]) successfully deleted
I20260619 12:39:25.940004 18478 log.cc:1196] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/ab06aa00abca478999b3cf9ab402593e
I20260619 12:39:25.940397 18478 ts_tablet_manager.cc:1939] T ab06aa00abca478999b3cf9ab402593e P 9fcbdad1030d442ebd1f996e3b9c154b: Deleting consensus metadata
W20260619 12:39:25.940784 17974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33147 (error 108)
I20260619 12:39:25.940982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 18179
I20260619 12:39:25.942420 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:25.942097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33147 (local address 127.0.0.1:55062)
0619 12:39:25.942233 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:25.942313 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 12:39:25.942636 17974 catalog_manager.cc:4584] TS c8eac9982e874252a0ccb8ba94682715 (127.0.98.1:33147): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab06aa00abca478999b3cf9ab402593e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33147: connect: Connection refused (error 111)
I20260619 12:39:25.943568 17975 catalog_manager.cc:4841] TS 9fcbdad1030d442ebd1f996e3b9c154b (127.0.98.3:41413): tablet ab06aa00abca478999b3cf9ab402593e (table test-table [id=5b096b624234473c800dc89d68af58d2]) successfully deleted
I20260619 12:39:25.947003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 18310
I20260619 12:39:25.952765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 17957
2026-06-19T12:39:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1252 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:26 dist-test-slave-kdrb krb5kdc[18483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:26 dist-test-slave-kdrb krb5kdc[18483](info): set up 2 sockets
Jun 19 12:39:26 dist-test-slave-kdrb krb5kdc[18483](info): commencing operation
krb5kdc: starting...
W20260619 12:39:26.999402 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:27 dist-test-slave-kdrb krb5kdc[18483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:39:27.126896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36987
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:27.244546 18496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:27.244850 18496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:27.244894 18496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:27.248994 18496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:27.249063 18496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:27.249089 18496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:27.249107 18496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:27.249122 18496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:27.254765 18496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:27.255954 18496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:27.256992 18496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:27.266177 18502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:27.266202 18504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:27.266185 18501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:27.266474 18496 server_base.cc:1034] running on GCE node
I20260619 12:39:27.267204 18496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:27.268447 18496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:27.269645 18496 hybrid_clock.cc:648] HybridClock initialized: now 1781872767269611 us; error 54 us; skew 500 ppm
Jun 19 12:39:27 dist-test-slave-kdrb krb5kdc[18483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:27.272635 18496 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:39:27.273677 18496 webserver.cc:460] Webserver started at http://127.0.98.62:34215/ using document root <none> and password file <none>
I20260619 12:39:27.274286 18496 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:27.274353 18496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:27.274581 18496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:27.276831 18496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "2241fc6611734b9bba5fc891aa6cdb32"
format_stamp: "Formatted at 2026-06-19 12:39:27 on dist-test-slave-kdrb"
server_key: "92185a33e087c2945b2b7027a7f3fc44"
server_key_iv: "b74686d775b34f9b760dd72cacc14721"
server_key_version: "encryptionkey@0"
I20260619 12:39:27.277432 18496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "2241fc6611734b9bba5fc891aa6cdb32"
format_stamp: "Formatted at 2026-06-19 12:39:27 on dist-test-slave-kdrb"
server_key: "92185a33e087c2945b2b7027a7f3fc44"
server_key_iv: "b74686d775b34f9b760dd72cacc14721"
server_key_version: "encryptionkey@0"
I20260619 12:39:27.281481 18496 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:39:27.284298 18511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:27.285676 18496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:27.285827 18496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "2241fc6611734b9bba5fc891aa6cdb32"
format_stamp: "Formatted at 2026-06-19 12:39:27 on dist-test-slave-kdrb"
server_key: "92185a33e087c2945b2b7027a7f3fc44"
server_key_iv: "b74686d775b34f9b760dd72cacc14721"
server_key_version: "encryptionkey@0"
I20260619 12:39:27.285951 18496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:27.295373 18496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:27.298517 18496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:27.298750 18496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:27.306454 18496 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36987
I20260619 12:39:27.306481 18563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36987 every 8 connection(s)
I20260619 12:39:27.307554 18496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:27.310463 18564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:27.314316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 18496
I20260619 12:39:27.314494 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:27.314826 392 external_mini_cluster.cc:1428] Setting key b8327019caade8be71015a0d8dd9d66e
I20260619 12:39:27.316913 18564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32: Bootstrap starting.
I20260619 12:39:27.319598 18564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:27.320391 18564 log.cc:824] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:27.322438 18564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32: No bootstrap required, opened a new log
Jun 19 12:39:27 dist-test-slave-kdrb krb5kdc[18483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:39:27.327178 18564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2241fc6611734b9bba5fc891aa6cdb32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36987 } }
I20260619 12:39:27.327476 18564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:27.327586 18564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2241fc6611734b9bba5fc891aa6cdb32, State: Initialized, Role: FOLLOWER
I20260619 12:39:27.328154 18564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2241fc6611734b9bba5fc891aa6cdb32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36987 } }
I20260619 12:39:27.328351 18564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:27.328447 18564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:27.328558 18564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:27.329560 18564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2241fc6611734b9bba5fc891aa6cdb32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36987 } }
I20260619 12:39:27.329988 18564 leader_election.cc:302] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2241fc6611734b9bba5fc891aa6cdb32; no voters:
I20260619 12:39:27.330358 18564 leader_election.cc:288] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:27.330487 18569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:27.330724 18569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [term 1 LEADER]: Becoming Leader. State: Replica: 2241fc6611734b9bba5fc891aa6cdb32, State: Running, Role: LEADER
I20260619 12:39:27.331058 18569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2241fc6611734b9bba5fc891aa6cdb32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36987 } }
I20260619 12:39:27.331192 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:27.316437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.0.98.62:36987)
0619 12:39:27.316879 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:27.316897 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:27.316933 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:39:27.317668 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:27.317687 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:27.318040 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:27.318340 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:27.319262 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:27.320486 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:27.321416 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:27.321922 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:27.322101 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:27.325359 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:27.325365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:27.325382 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:27.325418 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:27.327320 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:27.327992 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:27.327998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:27.328004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:27.328161 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:27.328514 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:27.328517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:27.328519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:27.329106 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:27.329363 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:27.329682 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 12:39:27.330015 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":155,"threads_started":1}
I20260619 12:39:27.331789 18564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:27.332917 18570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2241fc6611734b9bba5fc891aa6cdb32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2241fc6611734b9bba5fc891aa6cdb32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36987 } } }
I20260619 12:39:27.333060 18570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:27.333307 18570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2241fc6611734b9bba5fc891aa6cdb32. Latest consensus state: current_term: 1 leader_uuid: "2241fc6611734b9bba5fc891aa6cdb32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2241fc6611734b9bba5fc891aa6cdb32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36987 } } }
I20260619 12:39:27.333388 18570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32 [sys.catalog]: This master's current role is: LEADER
W20260619 12:39:27.337225 18584 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:39:27.337335 18584 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:39:27.337422 18578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:27.338464 18578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:27.345161 18578 catalog_manager.cc:1347] Generated new cluster ID: e0be551ce88c4879b556aeec3f6cd38a
I20260619 12:39:27.345261 18578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:27.380412 18578 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:27.381723 18578 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:27.392436 18578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32: Generated new TSK 0
I20260619 12:39:27.393280 18578 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:39:27.423517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36987
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:27.540428 18590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:27.540732 18590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:27.540786 18590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:27.545336 18590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:27.545426 18590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:27.545488 18590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:39:27.551115 18590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:27.552227 18590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:27.553242 18590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:27.563081 18595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:27.563086 18596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:27.563225 18590 server_base.cc:1034] running on GCE node
W20260619 12:39:27.563423 18598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:27.563930 18590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:27.564613 18590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:27.565836 18590 hybrid_clock.cc:648] HybridClock initialized: now 1781872767565764 us; error 92 us; skew 500 ppm
Jun 19 12:39:27 dist-test-slave-kdrb krb5kdc[18483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:27.568681 18590 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:39:27.569557 18590 webserver.cc:460] Webserver started at http://127.0.98.1:34869/ using document root <none> and password file <none>
I20260619 12:39:27.570133 18590 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:27.570225 18590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:27.570456 18590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:27.572633 18590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "9359bcc32cb64735bb1d111d043b8a20"
format_stamp: "Formatted at 2026-06-19 12:39:27 on dist-test-slave-kdrb"
server_key: "7e6ba9674394338b59624dc7a86a9474"
server_key_iv: "e19cb7682b03bdabd930c0fdc2536120"
server_key_version: "encryptionkey@0"
I20260619 12:39:27.573225 18590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9359bcc32cb64735bb1d111d043b8a20"
format_stamp: "Formatted at 2026-06-19 12:39:27 on dist-test-slave-kdrb"
server_key: "7e6ba9674394338b59624dc7a86a9474"
server_key_iv: "e19cb7682b03bdabd930c0fdc2536120"
server_key_version: "encryptionkey@0"
I20260619 12:39:27.577174 18590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:39:27.580052 18605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:27.581472 18590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:27.581660 18590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "9359bcc32cb64735bb1d111d043b8a20"
format_stamp: "Formatted at 2026-06-19 12:39:27 on dist-test-slave-kdrb"
server_key: "7e6ba9674394338b59624dc7a86a9474"
server_key_iv: "e19cb7682b03bdabd930c0fdc2536120"
server_key_version: "encryptionkey@0"
I20260619 12:39:27.581806 18590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:27.609580 18590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:27.613250 18590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:27.613508 18590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:27.614176 18590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:27.615079 18590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:27.615160 18590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:27.615237 18590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:27.615280 18590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:27.625070 18590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38923
I20260619 12:39:27.625145 18718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38923 every 8 connection(s)
I20260619 12:39:27.626071 18590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:39:27.630725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 18590
I20260619 12:39:27.630895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:39:27.631191 392 external_mini_cluster.cc:1428] Setting key 5441834d69be19a1734867ed8240be5e
Jun 19 12:39:27 dist-test-slave-kdrb krb5kdc[18483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:39:27.641855 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:27.628114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47823 (local address 127.0.98.62:36987)
0619 12:39:27.628254 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:27.628257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:27.629169 (+ 912us) server_negotiation.cc:407] Connection header received
0619 12:39:27.630047 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:27.630050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:27.630107 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:27.630186 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:27.631669 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:27.632550 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:27.633962 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:27.634388 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:27.634518 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:27.637675 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:27.637679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:27.637683 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:27.637717 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:27.639553 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:27.640110 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:27.640116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:27.640119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:27.640219 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:27.640582 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:27.640587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:27.640589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:27.640846 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:27.640960 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:27.641493 (+ 533us) server_negotiation.cc:299] Negotiation successful
0619 12:39:27.641620 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:39:27.642781 18721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:27.628503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36987 (local address 127.0.98.1:47823)
0619 12:39:27.628990 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:27.629035 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:27.629833 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:27.630361 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:27.630368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:27.630757 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:27.631461 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:27.631521 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:27.632760 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:27.632765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:27.633788 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:27.633826 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:27.634565 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:27.634570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:27.634833 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:27.635553 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:39:27.635583 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:27.637469 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:27.639686 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:27.639692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:27.639706 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:27.639994 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:27.640326 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:27.640329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:27.640330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:27.640480 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:27.640958 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:27.640963 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:27.641238 (+ 275us) client_negotiation.cc:769] Sending connection context
0619 12:39:27.641407 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 12:39:27.641709 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":116,"threads_started":1}
I20260619 12:39:27.644313 18719 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:39:27.644670 18719 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:27.645323 18719 heartbeater.cc:511] Master 127.0.98.62:36987 requested a full tablet report, sending...
I20260619 12:39:27.647287 18528 ts_manager.cc:194] Registered new tserver with Master: 9359bcc32cb64735bb1d111d043b8a20 (127.0.98.1:38923)
I20260619 12:39:27.649060 18528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:39:27.661245 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36987
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:27.796914 18724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:27.797201 18724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:27.797254 18724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:27.801554 18724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:27.801630 18724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:27.801726 18724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:39:27.807318 18724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:27.808573 18724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:27.809664 18724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:27.820087 18732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:27.820098 18729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:27.820104 18724 server_base.cc:1034] running on GCE node
W20260619 12:39:27.820168 18730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:27.820828 18724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:27.821509 18724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:27.822707 18724 hybrid_clock.cc:648] HybridClock initialized: now 1781872767822671 us; error 53 us; skew 500 ppm
Jun 19 12:39:27 dist-test-slave-kdrb krb5kdc[18483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:27.825716 18724 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:39:27.826655 18724 webserver.cc:460] Webserver started at http://127.0.98.2:41309/ using document root <none> and password file <none>
I20260619 12:39:27.827248 18724 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:27.827340 18724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:27.827631 18724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:27.829726 18724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "bd64152e7b0448b98f32cde6ee85b95b"
format_stamp: "Formatted at 2026-06-19 12:39:27 on dist-test-slave-kdrb"
server_key: "fcd2adfb4f07953bac359683e680b5eb"
server_key_iv: "c99b4f7904041cae322dc65712a93a3c"
server_key_version: "encryptionkey@0"
I20260619 12:39:27.830334 18724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bd64152e7b0448b98f32cde6ee85b95b"
format_stamp: "Formatted at 2026-06-19 12:39:27 on dist-test-slave-kdrb"
server_key: "fcd2adfb4f07953bac359683e680b5eb"
server_key_iv: "c99b4f7904041cae322dc65712a93a3c"
server_key_version: "encryptionkey@0"
I20260619 12:39:27.834592 18724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:39:27.837407 18739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:27.838991 18724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:39:27.839154 18724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "bd64152e7b0448b98f32cde6ee85b95b"
format_stamp: "Formatted at 2026-06-19 12:39:27 on dist-test-slave-kdrb"
server_key: "fcd2adfb4f07953bac359683e680b5eb"
server_key_iv: "c99b4f7904041cae322dc65712a93a3c"
server_key_version: "encryptionkey@0"
I20260619 12:39:27.839295 18724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:27.865581 18724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:27.869120 18724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:27.869335 18724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:27.869971 18724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:27.870838 18724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:27.870909 18724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:27.870981 18724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:27.871023 18724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:27.880312 18724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42033
I20260619 12:39:27.880328 18852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42033 every 8 connection(s)
I20260619 12:39:27.881291 18724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:39:27.881582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 18724
I20260619 12:39:27.881729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:39:27.882041 392 external_mini_cluster.cc:1428] Setting key d6f887d1652dbf11861fbca9ccaa9fc1
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Jun 19 12:39:27 dist-test-slave-kdrb krb5kdc[18483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:39:27.898818 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:27.883397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56121 (local address 127.0.98.62:36987)
0619 12:39:27.883573 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:27.883578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:27.884689 (+ 1111us) server_negotiation.cc:407] Connection header received
0619 12:39:27.885783 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:27.885786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:27.885841 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:27.885914 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:27.887692 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:27.888312 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:27.890356 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:27.890735 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:27.890833 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:27.894156 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:27.894163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:27.894170 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:27.894217 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:27.895896 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:27.896851 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:27.896855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:27.896857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:27.896981 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:27.897441 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:27.897448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:27.897453 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:27.897769 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:27.897897 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:27.898491 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 12:39:27.898642 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 12:39:27.899914 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:27.883992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36987 (local address 127.0.98.2:56121)
0619 12:39:27.884524 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:27.884569 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:27.885501 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:27.886136 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:27.886148 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:27.886773 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:27.887522 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:27.887547 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:27.888454 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:27.888458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:27.890181 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:27.890196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:27.890919 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:27.890925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:27.891161 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:27.891961 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:39:27.891988 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:27.893873 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:27.896124 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:27.896134 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:27.896162 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:27.896643 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:27.897137 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:27.897140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:27.897141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:27.897312 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:27.897910 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:27.897916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:27.898195 (+ 279us) client_negotiation.cc:769] Sending connection context
0619 12:39:27.898421 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 12:39:27.898823 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":143,"threads_started":1}
I20260619 12:39:27.901265 18853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36987
I20260619 12:39:27.901533 18853 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:27.902062 18853 heartbeater.cc:511] Master 127.0.98.62:36987 requested a full tablet report, sending...
I20260619 12:39:27.903196 18528 ts_manager.cc:194] Registered new tserver with Master: bd64152e7b0448b98f32cde6ee85b95b (127.0.98.2:42033)
I20260619 12:39:27.903858 18528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:39:27.915221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36987
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:28.031668 18858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:28.031956 18858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:28.032044 18858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:28.036597 18858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:28.036672 18858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:28.036736 18858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:39:28.042313 18858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:28.043751 18858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:28.044832 18858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:28.054747 18863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:28.054841 18864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:28.054747 18866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:28.055032 18858 server_base.cc:1034] running on GCE node
I20260619 12:39:28.055581 18858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:28.056279 18858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:28.057512 18858 hybrid_clock.cc:648] HybridClock initialized: now 1781872768057499 us; error 35 us; skew 500 ppm
Jun 19 12:39:28 dist-test-slave-kdrb krb5kdc[18483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:28.060508 18858 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:39:28.061421 18858 webserver.cc:460] Webserver started at http://127.0.98.3:38405/ using document root <none> and password file <none>
I20260619 12:39:28.062012 18858 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:28.062105 18858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:28.062343 18858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:28.064566 18858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "c598994630a2408cb1e168882df33988"
format_stamp: "Formatted at 2026-06-19 12:39:28 on dist-test-slave-kdrb"
server_key: "30e8e37b598827372cc76a2a1be1ff92"
server_key_iv: "7ae6b727cdfc5564c5e56916995edab8"
server_key_version: "encryptionkey@0"
I20260619 12:39:28.065156 18858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c598994630a2408cb1e168882df33988"
format_stamp: "Formatted at 2026-06-19 12:39:28 on dist-test-slave-kdrb"
server_key: "30e8e37b598827372cc76a2a1be1ff92"
server_key_iv: "7ae6b727cdfc5564c5e56916995edab8"
server_key_version: "encryptionkey@0"
I20260619 12:39:28.068939 18858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:39:28.071350 18873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:28.072595 18858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:39:28.072733 18858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "c598994630a2408cb1e168882df33988"
format_stamp: "Formatted at 2026-06-19 12:39:28 on dist-test-slave-kdrb"
server_key: "30e8e37b598827372cc76a2a1be1ff92"
server_key_iv: "7ae6b727cdfc5564c5e56916995edab8"
server_key_version: "encryptionkey@0"
I20260619 12:39:28.072847 18858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:28.092419 18858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:28.096119 18858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:28.096385 18858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:28.097168 18858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:28.098088 18858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:28.098173 18858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:28.098246 18858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:28.098299 18858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:28.108577 18858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34301
I20260619 12:39:28.108600 18986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34301 every 8 connection(s)
I20260619 12:39:28.109550 18858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:39:28.112670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 18858
I20260619 12:39:28.112818 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:39:28.113173 392 external_mini_cluster.cc:1428] Setting key 1ac2c95173a20d1d06ed400031cbd5b8
Jun 19 12:39:28 dist-test-slave-kdrb krb5kdc[18483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:39:28.124681 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.111428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55103 (local address 127.0.98.62:36987)
0619 12:39:28.111619 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:28.111624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:28.113049 (+ 1425us) server_negotiation.cc:407] Connection header received
0619 12:39:28.114384 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:28.114387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:28.114459 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:28.114556 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:28.115996 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.116537 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.117742 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.117966 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.118059 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.120603 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:28.120606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:28.120608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:28.120638 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:28.122283 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:28.122893 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:28.122896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:28.122898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:28.122979 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:28.123421 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:28.123424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:28.123425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:28.123702 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:28.123811 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:28.124407 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 12:39:28.124534 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 12:39:28.125591 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.111952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36987 (local address 127.0.98.3:55103)
0619 12:39:28.112842 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:28.112910 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:28.114170 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:28.114777 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:28.114786 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:28.115245 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:28.115865 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:28.115880 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.116657 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.116661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:28.117587 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:28.117596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.118112 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.118114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:28.118283 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.118910 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:39:28.118926 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:28.120430 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:28.122420 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:28.122425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:28.122439 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:28.122769 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:28.123104 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:28.123107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:28.123109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:28.123272 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:28.123832 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:28.123838 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:28.124125 (+ 287us) client_negotiation.cc:769] Sending connection context
0619 12:39:28.124295 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 12:39:28.124587 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":117,"threads_started":1}
I20260619 12:39:28.126919 18987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36987
I20260619 12:39:28.127231 18987 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:28.127859 18987 heartbeater.cc:511] Master 127.0.98.62:36987 requested a full tablet report, sending...
I20260619 12:39:28.128872 18528 ts_manager.cc:194] Registered new tserver with Master: c598994630a2408cb1e168882df33988 (127.0.98.3:34301)
I20260619 12:39:28.129441 18528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55103
I20260619 12:39:28.139832 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:39:28.150738 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.141943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.0.98.62:36987)
0619 12:39:28.142135 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:28.142140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:28.142268 (+ 128us) server_negotiation.cc:407] Connection header received
0619 12:39:28.142446 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:28.142449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:28.142493 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:28.142580 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:28.143736 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.144436 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.145310 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.145545 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.145660 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.147362 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:28.147366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:28.147369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:28.147408 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:28.148739 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:28.149315 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:28.149326 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:28.149328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:28.149407 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:28.149753 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:28.149757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:28.149759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:28.150007 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:28.150181 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:28.150457 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 12:39:28.150559 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 12:39:28.154132 18528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:39:28.157406 18528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:39:28.174031 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.167252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47480 (local address 127.0.98.1:38923)
0619 12:39:28.168053 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:28.168062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:28.168102 (+ 40us) server_negotiation.cc:407] Connection header received
0619 12:39:28.168194 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:28.168199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:28.168429 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:28.168991 (+ 562us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:28.169169 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.170516 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.172376 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.173127 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.173231 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.173390 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:28.173752 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 12:39:28.173860 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":117,"threads_started":1}
I20260619 12:39:28.174041 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.167087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38923 (local address 127.0.0.1:47480)
0619 12:39:28.167626 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:28.167686 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:28.167841 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:28.168618 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:28.168625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:28.168652 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:28.168807 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:28.168815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.170675 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.170679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:28.172207 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:28.172220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.173272 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.173276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:28.173551 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.173587 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 12:39:28.173715 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 12:39:28.173861 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":230,"thread_start_us":79,"threads_started":1}
I20260619 12:39:28.174762 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.167932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.0.98.2:42033)
0619 12:39:28.168468 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:28.168486 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:28.168513 (+ 27us) server_negotiation.cc:407] Connection header received
0619 12:39:28.168610 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:28.168616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:28.168812 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:28.169004 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:28.169288 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.170450 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.172139 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.173109 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.173232 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.173391 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:28.174474 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0619 12:39:28.174587 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":101,"threads_started":1}
I20260619 12:39:28.175196 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.167892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34301 (local address 127.0.0.1:42884)
0619 12:39:28.169150 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:28.169163 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:28.169265 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:28.169710 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:28.169713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:28.169730 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:28.169834 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:28.169840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.170947 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.170951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:28.172232 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:28.172239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.174793 (+ 2554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.174797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:28.174894 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.174914 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 12:39:28.174984 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 12:39:28.175057 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1198,"mutex_wait_us":52,"thread_start_us":96,"threads_started":1}
I20260619 12:39:28.175736 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.168211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.0.98.3:34301)
0619 12:39:28.168632 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:28.168643 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:28.169359 (+ 716us) server_negotiation.cc:407] Connection header received
0619 12:39:28.169428 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:28.169432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:28.169579 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:28.169728 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:28.169952 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.170812 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.173526 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.174434 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.174549 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.174728 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:28.175434 (+ 706us) server_negotiation.cc:299] Negotiation successful
0619 12:39:28.175584 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":119,"threads_started":1}
I20260619 12:39:28.174098 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.167159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42033 (local address 127.0.0.1:38562)
0619 12:39:28.168105 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:28.168121 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:28.168201 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:28.169009 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:28.169012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:28.169036 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:28.169143 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:28.169150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.170595 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.170598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:28.172012 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:28.172021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.173272 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.173276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:28.173508 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.173583 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 12:39:28.173703 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 12:39:28.173855 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":43,"thread_start_us":93,"threads_started":1}
I20260619 12:39:28.177806 18653 tablet_service.cc:1467] Processing CreateTablet for tablet 06c64f104e6046b5907eeafe56d4c5e8 (DEFAULT_TABLE table=test-table [id=99492f019db7407db857dac413130ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:28.179052 18653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06c64f104e6046b5907eeafe56d4c5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:28.178826 18787 tablet_service.cc:1467] Processing CreateTablet for tablet 06c64f104e6046b5907eeafe56d4c5e8 (DEFAULT_TABLE table=test-table [id=99492f019db7407db857dac413130ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:28.178855 18921 tablet_service.cc:1467] Processing CreateTablet for tablet 06c64f104e6046b5907eeafe56d4c5e8 (DEFAULT_TABLE table=test-table [id=99492f019db7407db857dac413130ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:28.179863 18787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06c64f104e6046b5907eeafe56d4c5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:28.179901 18921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06c64f104e6046b5907eeafe56d4c5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:28.186681 19004 tablet_bootstrap.cc:492] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b: Bootstrap starting.
I20260619 12:39:28.187491 19005 tablet_bootstrap.cc:492] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20: Bootstrap starting.
I20260619 12:39:28.189121 19006 tablet_bootstrap.cc:492] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988: Bootstrap starting.
I20260619 12:39:28.189554 19005 tablet_bootstrap.cc:654] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:28.189939 19004 tablet_bootstrap.cc:654] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:28.190506 19005 log.cc:824] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:28.190711 19004 log.cc:824] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:28.191496 19006 tablet_bootstrap.cc:654] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:28.192802 19006 log.cc:824] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:28.192993 19005 tablet_bootstrap.cc:492] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20: No bootstrap required, opened a new log
I20260619 12:39:28.193222 19005 ts_tablet_manager.cc:1397] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260619 12:39:28.194626 19006 tablet_bootstrap.cc:492] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988: No bootstrap required, opened a new log
I20260619 12:39:28.194679 19004 tablet_bootstrap.cc:492] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b: No bootstrap required, opened a new log
I20260619 12:39:28.194875 19004 ts_tablet_manager.cc:1397] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260619 12:39:28.194988 19006 ts_tablet_manager.cc:1397] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:39:28.198009 19006 raft_consensus.cc:348] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9359bcc32cb64735bb1d111d043b8a20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38923 } } peers { permanent_uuid: "bd64152e7b0448b98f32cde6ee85b95b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } } peers { permanent_uuid: "c598994630a2408cb1e168882df33988" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34301 } }
I20260619 12:39:28.198027 19005 raft_consensus.cc:348] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9359bcc32cb64735bb1d111d043b8a20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38923 } } peers { permanent_uuid: "bd64152e7b0448b98f32cde6ee85b95b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } } peers { permanent_uuid: "c598994630a2408cb1e168882df33988" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34301 } }
I20260619 12:39:28.198311 19005 raft_consensus.cc:374] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:28.198312 19006 raft_consensus.cc:374] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:28.198374 19005 raft_consensus.cc:729] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9359bcc32cb64735bb1d111d043b8a20, State: Initialized, Role: FOLLOWER
I20260619 12:39:28.198382 19006 raft_consensus.cc:729] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c598994630a2408cb1e168882df33988, State: Initialized, Role: FOLLOWER
I20260619 12:39:28.198843 19006 consensus_queue.cc:260] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9359bcc32cb64735bb1d111d043b8a20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38923 } } peers { permanent_uuid: "bd64152e7b0448b98f32cde6ee85b95b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } } peers { permanent_uuid: "c598994630a2408cb1e168882df33988" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34301 } }
I20260619 12:39:28.198854 19005 consensus_queue.cc:260] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9359bcc32cb64735bb1d111d043b8a20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38923 } } peers { permanent_uuid: "bd64152e7b0448b98f32cde6ee85b95b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } } peers { permanent_uuid: "c598994630a2408cb1e168882df33988" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34301 } }
I20260619 12:39:28.198865 19004 raft_consensus.cc:348] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9359bcc32cb64735bb1d111d043b8a20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38923 } } peers { permanent_uuid: "bd64152e7b0448b98f32cde6ee85b95b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } } peers { permanent_uuid: "c598994630a2408cb1e168882df33988" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34301 } }
I20260619 12:39:28.199111 19004 raft_consensus.cc:374] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:28.199182 19004 raft_consensus.cc:729] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd64152e7b0448b98f32cde6ee85b95b, State: Initialized, Role: FOLLOWER
I20260619 12:39:28.199626 18987 heartbeater.cc:503] Master 127.0.98.62:36987 was elected leader, sending a full tablet report...
I20260619 12:39:28.199617 19004 consensus_queue.cc:260] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9359bcc32cb64735bb1d111d043b8a20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38923 } } peers { permanent_uuid: "bd64152e7b0448b98f32cde6ee85b95b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } } peers { permanent_uuid: "c598994630a2408cb1e168882df33988" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34301 } }
I20260619 12:39:28.199946 19006 ts_tablet_manager.cc:1428] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:39:28.199957 19005 ts_tablet_manager.cc:1428] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260619 12:39:28.200335 18719 heartbeater.cc:503] Master 127.0.98.62:36987 was elected leader, sending a full tablet report...
I20260619 12:39:28.200495 19004 ts_tablet_manager.cc:1428] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 12:39:28.200567 18853 heartbeater.cc:503] Master 127.0.98.62:36987 was elected leader, sending a full tablet report...
I20260619 12:39:28.235275 19011 raft_consensus.cc:482] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:39:28.235602 19011 raft_consensus.cc:504] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9359bcc32cb64735bb1d111d043b8a20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38923 } } peers { permanent_uuid: "bd64152e7b0448b98f32cde6ee85b95b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } } peers { permanent_uuid: "c598994630a2408cb1e168882df33988" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34301 } }
I20260619 12:39:28.237003 19011 leader_election.cc:288] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd64152e7b0448b98f32cde6ee85b95b (127.0.98.2:42033), c598994630a2408cb1e168882df33988 (127.0.98.3:34301)
I20260619 12:39:28.242748 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.237391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53341 (local address 127.0.98.3:34301)
0619 12:39:28.237528 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:28.237533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:28.237866 (+ 333us) server_negotiation.cc:407] Connection header received
0619 12:39:28.238021 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:28.238024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:28.238077 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:28.238168 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:28.238597 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.239386 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.241332 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.242031 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.242131 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.242205 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:28.242500 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 12:39:28.242596 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 12:39:28.242741 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.237405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43451 (local address 127.0.98.2:42033)
0619 12:39:28.237546 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:28.237551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:28.237865 (+ 314us) server_negotiation.cc:407] Connection header received
0619 12:39:28.238021 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:28.238024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:28.238077 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:28.238168 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:28.238597 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.239386 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.241364 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.242009 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.242131 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.242207 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:28.242500 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 12:39:28.242596 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:39:28.242925 19014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.237277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42033 (local address 127.0.98.1:43451)
0619 12:39:28.237758 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:28.237773 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:28.237894 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:28.238202 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:28.238205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:28.238232 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:28.238474 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:28.238482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.239545 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.239549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:28.241176 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:28.241190 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.242206 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.242210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:28.242330 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.242376 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 12:39:28.242470 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 12:39:28.242707 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":337,"spinlock_wait_cycles":28032,"thread_start_us":126,"threads_started":1}
I20260619 12:39:28.242925 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.237277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34301 (local address 127.0.98.1:53341)
0619 12:39:28.237737 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:28.237753 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:28.237894 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:28.238202 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:28.238205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:28.238234 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:28.238474 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:28.238482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.239545 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.239549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:28.241176 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:28.241190 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.242243 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.242246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:28.242341 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.242377 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 12:39:28.242470 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 12:39:28.242707 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":233,"spinlock_wait_cycles":18432,"thread_start_us":136,"threads_started":1}
I20260619 12:39:28.243809 18941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06c64f104e6046b5907eeafe56d4c5e8" candidate_uuid: "9359bcc32cb64735bb1d111d043b8a20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c598994630a2408cb1e168882df33988" is_pre_election: true
I20260619 12:39:28.243878 18807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06c64f104e6046b5907eeafe56d4c5e8" candidate_uuid: "9359bcc32cb64735bb1d111d043b8a20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd64152e7b0448b98f32cde6ee85b95b" is_pre_election: true
I20260619 12:39:28.244165 18941 raft_consensus.cc:2454] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9359bcc32cb64735bb1d111d043b8a20 in term 0.
I20260619 12:39:28.244242 18807 raft_consensus.cc:2454] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9359bcc32cb64735bb1d111d043b8a20 in term 0.
I20260619 12:39:28.244796 18609 leader_election.cc:302] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9359bcc32cb64735bb1d111d043b8a20, bd64152e7b0448b98f32cde6ee85b95b, c598994630a2408cb1e168882df33988; no voters:
I20260619 12:39:28.245105 19011 raft_consensus.cc:2781] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:39:28.245187 19011 raft_consensus.cc:482] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:39:28.245245 19011 raft_consensus.cc:3037] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:28.246317 19011 raft_consensus.cc:504] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9359bcc32cb64735bb1d111d043b8a20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38923 } } peers { permanent_uuid: "bd64152e7b0448b98f32cde6ee85b95b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } } peers { permanent_uuid: "c598994630a2408cb1e168882df33988" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34301 } }
I20260619 12:39:28.246729 19011 leader_election.cc:288] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [CANDIDATE]: Term 1 election: Requested vote from peers bd64152e7b0448b98f32cde6ee85b95b (127.0.98.2:42033), c598994630a2408cb1e168882df33988 (127.0.98.3:34301)
I20260619 12:39:28.247152 18807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06c64f104e6046b5907eeafe56d4c5e8" candidate_uuid: "9359bcc32cb64735bb1d111d043b8a20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd64152e7b0448b98f32cde6ee85b95b"
I20260619 12:39:28.247268 18941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06c64f104e6046b5907eeafe56d4c5e8" candidate_uuid: "9359bcc32cb64735bb1d111d043b8a20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c598994630a2408cb1e168882df33988"
I20260619 12:39:28.247347 18807 raft_consensus.cc:3037] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:28.247419 18941 raft_consensus.cc:3037] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:28.248534 18941 raft_consensus.cc:2454] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9359bcc32cb64735bb1d111d043b8a20 in term 1.
I20260619 12:39:28.248548 18807 raft_consensus.cc:2454] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9359bcc32cb64735bb1d111d043b8a20 in term 1.
I20260619 12:39:28.248860 18609 leader_election.cc:302] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9359bcc32cb64735bb1d111d043b8a20, c598994630a2408cb1e168882df33988; no voters:
I20260619 12:39:28.249070 19011 raft_consensus.cc:2781] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:28.249315 19011 raft_consensus.cc:686] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [term 1 LEADER]: Becoming Leader. State: Replica: 9359bcc32cb64735bb1d111d043b8a20, State: Running, Role: LEADER
I20260619 12:39:28.249665 19011 consensus_queue.cc:237] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9359bcc32cb64735bb1d111d043b8a20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38923 } } peers { permanent_uuid: "bd64152e7b0448b98f32cde6ee85b95b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } } peers { permanent_uuid: "c598994630a2408cb1e168882df33988" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34301 } }
I20260619 12:39:28.253057 18527 catalog_manager.cc:5495] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9359bcc32cb64735bb1d111d043b8a20 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9359bcc32cb64735bb1d111d043b8a20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9359bcc32cb64735bb1d111d043b8a20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd64152e7b0448b98f32cde6ee85b95b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c598994630a2408cb1e168882df33988" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34301 } health_report { overall_health: UNKNOWN } } }
I20260619 12:39:28.280615 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.277121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47496 (local address 127.0.98.1:38923)
0619 12:39:28.277273 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:28.277277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:28.277291 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:39:28.277363 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:28.277365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:28.277421 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:28.277516 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:39:28.277743 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.278279 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.279139 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.279359 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.279438 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.279769 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:39:28.280207 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:39:28.280328 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:28.280464 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 12:39:28.280500 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 12:39:28.286959 18941 raft_consensus.cc:1261] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988 [term 1 FOLLOWER]: Refusing update from remote peer 9359bcc32cb64735bb1d111d043b8a20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:39:28.286989 18807 raft_consensus.cc:1261] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b [term 1 FOLLOWER]: Refusing update from remote peer 9359bcc32cb64735bb1d111d043b8a20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:39:28.287757 19015 consensus_queue.cc:1035] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c598994630a2408cb1e168882df33988" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:39:28.287958 19011 consensus_queue.cc:1035] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd64152e7b0448b98f32cde6ee85b95b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:39:28.297581 19019 mvcc.cc:205] Tried to move back new op lower bound from 7298550858894884864 to 7298550858753519616. Current Snapshot: MvccSnapshot[applied={T|T < 7298550858894884864}]
I20260619 12:39:28.305547 18526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:39:28.305751 18526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:39:28.308358 18526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2241fc6611734b9bba5fc891aa6cdb32: Sending DeleteTablet for 3 replicas of tablet 06c64f104e6046b5907eeafe56d4c5e8
I20260619 12:39:28.309350 18787 tablet_service.cc:1514] Processing DeleteTablet for tablet 06c64f104e6046b5907eeafe56d4c5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38562
I20260619 12:39:28.309511 18653 tablet_service.cc:1514] Processing DeleteTablet for tablet 06c64f104e6046b5907eeafe56d4c5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47480
I20260619 12:39:28.309952 18921 tablet_service.cc:1514] Processing DeleteTablet for tablet 06c64f104e6046b5907eeafe56d4c5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42884
I20260619 12:39:28.310400 19029 tablet_replica.cc:330] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988: stopping tablet replica
I20260619 12:39:28.310737 19029 raft_consensus.cc:2229] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:39:28.311028 19028 tablet_replica.cc:330] T 06c64f104e6046b5907eeafe56d4c5e8 P 9359bcc32cb64735bb1d111d043b8a20: stopping tablet replica
I20260619 12:39:28.311311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 18590
I20260619 12:39:28.313192 19027 tablet_replica.cc:330] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b: stopping tablet replica
I20260619 12:39:28.313557 19027 raft_consensus.cc:2229] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:39:28.313934 19027 raft_consensus.cc:2258] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:39:28.315608 19027 ts_tablet_manager.cc:1905] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:39:28.315969 19029 raft_consensus.cc:2258] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:39:28.317535 19029 ts_tablet_manager.cc:1905] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:39:28.319262 19027 ts_tablet_manager.cc:1918] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:39:28.319375 19027 log.cc:1196] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/06c64f104e6046b5907eeafe56d4c5e8
I20260619 12:39:28.319821 19027 ts_tablet_manager.cc:1939] T 06c64f104e6046b5907eeafe56d4c5e8 P bd64152e7b0448b98f32cde6ee85b95b: Deleting consensus metadata
W20260619 12:39:28.319948 18515 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38923 (error 108)
I20260619 12:39:28.320829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 18724
I20260619 12:39:28.320876 19029 ts_tablet_manager.cc:1918] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:39:28.321198 19029 log.cc:1196] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/06c64f104e6046b5907eeafe56d4c5e8
I20260619 12:39:28.321584 19029 ts_tablet_manager.cc:1939] T 06c64f104e6046b5907eeafe56d4c5e8 P c598994630a2408cb1e168882df33988: Deleting consensus metadata
I20260619 12:39:28.322927 18515 catalog_manager.cc:4841] TS c598994630a2408cb1e168882df33988 (127.0.98.3:34301): tablet 06c64f104e6046b5907eeafe56d4c5e8 (table test-table [id=99492f019db7407db857dac413130ad0]) successfully deleted
I20260619 12:39:28.323746 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.323313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38923 (local address 127.0.0.1:47512)
0619 12:39:28.323548 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:28.323625 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260619 12:39:28.323904 18515 catalog_manager.cc:4584] TS 9359bcc32cb64735bb1d111d043b8a20 (127.0.98.1:38923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06c64f104e6046b5907eeafe56d4c5e8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38923: connect: Connection refused (error 111)
I20260619 12:39:28.328123 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 18858
I20260619 12:39:28.329226 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.328338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42033 (local address 127.0.0.1:38568)
0619 12:39:28.329041 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:28.329122 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598}
I20260619 12:39:28.334165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 18496
2026-06-19T12:39:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2383 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T12:39:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:28Z Disabled control of system clock
I20260619 12:39:28.365666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44523
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:39:28.481073 19034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:28.481387 19034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:28.481459 19034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:28.485737 19034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:28.485824 19034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:28.485841 19034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:28.485857 19034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:28.485872 19034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:28.491155 19034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44523
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:28.492379 19034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:28.493834 19034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:28.502876 19040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:28.502933 19034 server_base.cc:1034] running on GCE node
W20260619 12:39:28.502863 19039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:28.502890 19042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:28.503787 19034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:28.504941 19034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:28.506134 19034 hybrid_clock.cc:648] HybridClock initialized: now 1781872768506112 us; error 33 us; skew 500 ppm
I20260619 12:39:28.508107 19034 webserver.cc:460] Webserver started at http://127.0.98.62:33555/ using document root <none> and password file <none>
I20260619 12:39:28.508723 19034 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:28.508817 19034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:28.509059 19034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:28.511238 19034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "0eabd9bc4e5d40a8ab7004d974489102"
format_stamp: "Formatted at 2026-06-19 12:39:28 on dist-test-slave-kdrb"
server_key: "e4d2a97030799efa18c070250002c6e0"
server_key_iv: "69153b5243f990bc5c019a97090bf45c"
server_key_version: "encryptionkey@0"
I20260619 12:39:28.511878 19034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "0eabd9bc4e5d40a8ab7004d974489102"
format_stamp: "Formatted at 2026-06-19 12:39:28 on dist-test-slave-kdrb"
server_key: "e4d2a97030799efa18c070250002c6e0"
server_key_iv: "69153b5243f990bc5c019a97090bf45c"
server_key_version: "encryptionkey@0"
I20260619 12:39:28.515625 19034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:39:28.518100 19048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:28.519335 19034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:39:28.519543 19034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "0eabd9bc4e5d40a8ab7004d974489102"
format_stamp: "Formatted at 2026-06-19 12:39:28 on dist-test-slave-kdrb"
server_key: "e4d2a97030799efa18c070250002c6e0"
server_key_iv: "69153b5243f990bc5c019a97090bf45c"
server_key_version: "encryptionkey@0"
I20260619 12:39:28.519667 19034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:28.540668 19034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:28.541463 19034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:28.541724 19034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:28.551327 19034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44523
I20260619 12:39:28.551362 19100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44523 every 8 connection(s)
I20260619 12:39:28.552603 19034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:28.555876 19101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:28.561537 19101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102: Bootstrap starting.
I20260619 12:39:28.562649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 19034
I20260619 12:39:28.562752 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:28.563053 392 external_mini_cluster.cc:1428] Setting key cef8835a1a53b4d032ea5a0f2a28ecca
I20260619 12:39:28.564460 19101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:28.565232 19101 log.cc:824] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:28.567373 19101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102: No bootstrap required, opened a new log
I20260619 12:39:28.571797 19101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eabd9bc4e5d40a8ab7004d974489102" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } }
I20260619 12:39:28.572060 19101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:28.572141 19101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eabd9bc4e5d40a8ab7004d974489102, State: Initialized, Role: FOLLOWER
I20260619 12:39:28.572451 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.564301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58350 (local address 127.0.98.62:44523)
0619 12:39:28.564980 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:28.564995 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:28.565030 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:39:28.565738 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:28.565766 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:28.566105 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:28.566397 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:28.566733 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.567809 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.569006 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.569481 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.569604 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.570476 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:28.570481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:28.570495 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:28.570893 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:28.570950 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:28.570970 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:28.571047 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:28.571199 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 12:39:28.571553 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":141,"threads_started":1}
I20260619 12:39:28.572680 19101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eabd9bc4e5d40a8ab7004d974489102" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } }
I20260619 12:39:28.572836 19101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:28.572901 19101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:28.572993 19101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:28.573999 19101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eabd9bc4e5d40a8ab7004d974489102" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } }
I20260619 12:39:28.574342 19101 leader_election.cc:302] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eabd9bc4e5d40a8ab7004d974489102; no voters:
I20260619 12:39:28.574677 19101 leader_election.cc:288] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:28.574775 19106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:28.575011 19106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [term 1 LEADER]: Becoming Leader. State: Replica: 0eabd9bc4e5d40a8ab7004d974489102, State: Running, Role: LEADER
I20260619 12:39:28.575315 19106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eabd9bc4e5d40a8ab7004d974489102" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } }
I20260619 12:39:28.575873 19101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:28.577129 19108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eabd9bc4e5d40a8ab7004d974489102. Latest consensus state: current_term: 1 leader_uuid: "0eabd9bc4e5d40a8ab7004d974489102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eabd9bc4e5d40a8ab7004d974489102" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } } }
I20260619 12:39:28.577251 19108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:28.577735 19115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:28.578007 19107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eabd9bc4e5d40a8ab7004d974489102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eabd9bc4e5d40a8ab7004d974489102" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } } }
I20260619 12:39:28.578140 19107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:28.580508 19115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:28.586462 19115 catalog_manager.cc:1347] Generated new cluster ID: dc34402b84ce476b983bebaa9d0f966c
I20260619 12:39:28.586781 19115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:28.598920 19115 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:28.600203 19115 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:28.612274 19115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102: Generated new TSK 0
I20260619 12:39:28.613161 19115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:28.616117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--builtin_ntp_servers=127.0.98.20:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:39:28.729969 19125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:28.730253 19125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:28.730293 19125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:28.734474 19125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:28.734540 19125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:28.734598 19125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:39:28.739281 19125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:28.740446 19125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:28.741886 19125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:28.751705 19130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:28.751756 19133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:28.751775 19131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:28.751870 19125 server_base.cc:1034] running on GCE node
I20260619 12:39:28.752358 19125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:28.753036 19125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:28.754231 19125 hybrid_clock.cc:648] HybridClock initialized: now 1781872768754199 us; error 50 us; skew 500 ppm
I20260619 12:39:28.760350 19125 webserver.cc:460] Webserver started at http://127.0.98.1:35147/ using document root <none> and password file <none>
I20260619 12:39:28.760988 19125 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:28.761055 19125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:28.761276 19125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:28.763571 19125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "48474812788049fba7c40dfb143688e2"
format_stamp: "Formatted at 2026-06-19 12:39:28 on dist-test-slave-kdrb"
server_key: "a50a5e898f15f281a4b8721e951049c4"
server_key_iv: "2c5ddbcb63cf10b23cf7cd0f42842d0c"
server_key_version: "encryptionkey@0"
I20260619 12:39:28.764173 19125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "48474812788049fba7c40dfb143688e2"
format_stamp: "Formatted at 2026-06-19 12:39:28 on dist-test-slave-kdrb"
server_key: "a50a5e898f15f281a4b8721e951049c4"
server_key_iv: "2c5ddbcb63cf10b23cf7cd0f42842d0c"
server_key_version: "encryptionkey@0"
I20260619 12:39:28.768029 19125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 12:39:28.770574 19139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:28.772024 19125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:28.772171 19125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "48474812788049fba7c40dfb143688e2"
format_stamp: "Formatted at 2026-06-19 12:39:28 on dist-test-slave-kdrb"
server_key: "a50a5e898f15f281a4b8721e951049c4"
server_key_iv: "2c5ddbcb63cf10b23cf7cd0f42842d0c"
server_key_version: "encryptionkey@0"
I20260619 12:39:28.772303 19125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:28.793475 19125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:28.794242 19125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:28.794472 19125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:28.795107 19125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:28.796124 19125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:28.796196 19125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:28.796273 19125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:28.796322 19125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:28.805312 19125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40925
I20260619 12:39:28.805331 19252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40925 every 8 connection(s)
I20260619 12:39:28.806267 19125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:39:28.812705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 19125
I20260619 12:39:28.812958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:39:28.813261 392 external_mini_cluster.cc:1428] Setting key 8f2074a3a53fd8ab8e925834bf3a63ee
I20260619 12:39:28.815402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--builtin_ntp_servers=127.0.98.20:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:39:28.816442 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.808231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33025 (local address 127.0.98.62:44523)
0619 12:39:28.808368 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:28.808372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:28.809437 (+ 1065us) server_negotiation.cc:407] Connection header received
0619 12:39:28.810902 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:28.810905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:28.810996 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:28.811108 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:28.812149 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.812716 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.814244 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.814528 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.814636 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.815924 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:28.815927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:28.815930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:28.815951 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:28.815971 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:28.815979 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:28.816060 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:28.816232 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 12:39:28.816303 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 12:39:28.817337 19255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:28.808601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44523 (local address 127.0.98.1:33025)
0619 12:39:28.809274 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:28.809317 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:28.810664 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:28.811301 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:28.811309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:28.811401 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:28.811992 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:28.812006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.812854 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.812857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:28.814097 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:28.814108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:28.814705 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:28.814710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:28.814923 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:28.815634 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:39:28.815636 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:28.815680 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:39:28.815758 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:28.816061 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:28.816078 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:28.816087 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 12:39:28.816232 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 12:39:28.816548 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260619 12:39:28.818562 19253 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44523
I20260619 12:39:28.818823 19253 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:28.819368 19253 heartbeater.cc:511] Master 127.0.98.62:44523 requested a full tablet report, sending...
I20260619 12:39:28.820911 19063 ts_manager.cc:194] Registered new tserver with Master: 48474812788049fba7c40dfb143688e2 (127.0.98.1:40925)
I20260619 12:39:28.822079 19063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:33025
W20260619 12:39:28.929553 19256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:28.929874 19256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:28.929961 19256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:28.934249 19256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:28.934353 19256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:28.934455 19256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:39:28.939378 19256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:28.940550 19256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:28.942049 19256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:28.951987 19261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:28.952103 19262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:28.952083 19256 server_base.cc:1034] running on GCE node
W20260619 12:39:28.951987 19264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:28.952674 19256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:28.953336 19256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:28.954519 19256 hybrid_clock.cc:648] HybridClock initialized: now 1781872768954490 us; error 41 us; skew 500 ppm
I20260619 12:39:28.956446 19256 webserver.cc:460] Webserver started at http://127.0.98.2:37865/ using document root <none> and password file <none>
I20260619 12:39:28.957038 19256 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:28.957129 19256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:28.957374 19256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:28.959414 19256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "7057a7a252c44602935edf3196d79e62"
format_stamp: "Formatted at 2026-06-19 12:39:28 on dist-test-slave-kdrb"
server_key: "3693234082002cbda414a55a22089087"
server_key_iv: "9badf88430c426cb10f111f8232f61a8"
server_key_version: "encryptionkey@0"
I20260619 12:39:28.960027 19256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7057a7a252c44602935edf3196d79e62"
format_stamp: "Formatted at 2026-06-19 12:39:28 on dist-test-slave-kdrb"
server_key: "3693234082002cbda414a55a22089087"
server_key_iv: "9badf88430c426cb10f111f8232f61a8"
server_key_version: "encryptionkey@0"
I20260619 12:39:28.963759 19256 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:39:28.966197 19270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:28.967427 19256 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:39:28.967607 19256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "7057a7a252c44602935edf3196d79e62"
format_stamp: "Formatted at 2026-06-19 12:39:28 on dist-test-slave-kdrb"
server_key: "3693234082002cbda414a55a22089087"
server_key_iv: "9badf88430c426cb10f111f8232f61a8"
server_key_version: "encryptionkey@0"
I20260619 12:39:28.967725 19256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:28.998051 19256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:28.998822 19256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:28.999030 19256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:28.999732 19256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:29.000700 19256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:29.000748 19256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:29.000788 19256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:29.000803 19256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:29.010249 19256 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35001
I20260619 12:39:29.010270 19383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35001 every 8 connection(s)
I20260619 12:39:29.011217 19256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:39:29.012352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 19256
I20260619 12:39:29.012452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:39:29.012782 392 external_mini_cluster.cc:1428] Setting key 1cb9096aa82a06978e3e8f700822baad
I20260619 12:39:29.014981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--builtin_ntp_servers=127.0.98.20:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:39:29.021307 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.013301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36733 (local address 127.0.98.62:44523)
0619 12:39:29.013461 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:29.013467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:29.014817 (+ 1350us) server_negotiation.cc:407] Connection header received
0619 12:39:29.016373 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:29.016377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:29.016433 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:29.016537 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:29.017477 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.018011 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.019189 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.019423 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.019606 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:29.020773 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:29.020776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:29.020779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:29.020794 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:29.020829 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:29.020838 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:29.020915 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:29.021081 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 12:39:29.021147 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 12:39:29.022186 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.013854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44523 (local address 127.0.98.2:36733)
0619 12:39:29.014629 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:29.014685 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:29.016144 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:29.016725 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:29.016732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:29.016827 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:29.017327 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:29.017339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.018155 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.018163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:29.019058 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:29.019067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.019659 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.019664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:29.019839 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:29.020494 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:39:29.020496 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:29.020539 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:39:29.020607 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:29.020929 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:29.020942 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:29.020952 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 12:39:29.021083 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 12:39:29.021383 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":143,"threads_started":1}
I20260619 12:39:29.023393 19384 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44523
I20260619 12:39:29.023676 19384 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:29.024248 19384 heartbeater.cc:511] Master 127.0.98.62:44523 requested a full tablet report, sending...
I20260619 12:39:29.025339 19063 ts_manager.cc:194] Registered new tserver with Master: 7057a7a252c44602935edf3196d79e62 (127.0.98.2:35001)
I20260619 12:39:29.026017 19063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36733
W20260619 12:39:29.129662 19387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:29.129943 19387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:29.129985 19387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:29.134087 19387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:29.134155 19387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:29.134213 19387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:39:29.139030 19387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:29.140187 19387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:29.141592 19387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:29.152179 19395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:29.152154 19393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:29.152621 19387 server_base.cc:1034] running on GCE node
W20260619 12:39:29.152153 19392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:29.153157 19387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:29.153846 19387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:29.155077 19387 hybrid_clock.cc:648] HybridClock initialized: now 1781872769155037 us; error 53 us; skew 500 ppm
I20260619 12:39:29.157279 19387 webserver.cc:460] Webserver started at http://127.0.98.3:43849/ using document root <none> and password file <none>
I20260619 12:39:29.157943 19387 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:29.158038 19387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:29.158281 19387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:29.160566 19387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "868d9a8a39e44889b109ad0e8adf748b"
format_stamp: "Formatted at 2026-06-19 12:39:29 on dist-test-slave-kdrb"
server_key: "695eec3fdd1ba058432a5abba9fcc1c7"
server_key_iv: "d77d84fcd6421761e0f3880eaa12c0b9"
server_key_version: "encryptionkey@0"
I20260619 12:39:29.161209 19387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "868d9a8a39e44889b109ad0e8adf748b"
format_stamp: "Formatted at 2026-06-19 12:39:29 on dist-test-slave-kdrb"
server_key: "695eec3fdd1ba058432a5abba9fcc1c7"
server_key_iv: "d77d84fcd6421761e0f3880eaa12c0b9"
server_key_version: "encryptionkey@0"
I20260619 12:39:29.165221 19387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 12:39:29.167887 19401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:29.169224 19387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 12:39:29.169420 19387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "868d9a8a39e44889b109ad0e8adf748b"
format_stamp: "Formatted at 2026-06-19 12:39:29 on dist-test-slave-kdrb"
server_key: "695eec3fdd1ba058432a5abba9fcc1c7"
server_key_iv: "d77d84fcd6421761e0f3880eaa12c0b9"
server_key_version: "encryptionkey@0"
I20260619 12:39:29.169564 19387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:29.195291 19387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:29.196123 19387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:29.196344 19387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:29.196976 19387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:29.197968 19387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:29.198040 19387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:29.198113 19387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:29.198158 19387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:29.207460 19387 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46427
I20260619 12:39:29.207553 19514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46427 every 8 connection(s)
I20260619 12:39:29.208467 19387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:39:29.211642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 19387
I20260619 12:39:29.211755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:39:29.212097 392 external_mini_cluster.cc:1428] Setting key 4374c615f7318a726900709183d6ebed
I20260619 12:39:29.218556 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.210556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41687 (local address 127.0.98.62:44523)
0619 12:39:29.210736 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:29.210741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:29.212036 (+ 1295us) server_negotiation.cc:407] Connection header received
0619 12:39:29.213723 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:29.213728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:29.213785 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:29.213866 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:29.214974 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.215578 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.216741 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.216976 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.217068 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:29.218129 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:29.218132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:29.218134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:29.218147 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:29.218169 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:29.218177 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:29.218252 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:29.218384 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 12:39:29.218439 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 12:39:29.219432 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.210980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44523 (local address 127.0.98.3:41687)
0619 12:39:29.211839 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:29.211899 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:29.213501 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:29.214084 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:29.214094 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:29.214210 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:29.214837 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:29.214853 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.215703 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.215706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:29.216612 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:29.216622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.217117 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.217120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:29.217259 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:29.217861 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:39:29.217863 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:29.217904 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:39:29.217972 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:29.218257 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:29.218263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:29.218267 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:39:29.218366 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 12:39:29.218591 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
I20260619 12:39:29.220652 19515 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44523
I20260619 12:39:29.220883 19515 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:29.221367 19515 heartbeater.cc:511] Master 127.0.98.62:44523 requested a full tablet report, sending...
I20260619 12:39:29.222386 19063 ts_manager.cc:194] Registered new tserver with Master: 868d9a8a39e44889b109ad0e8adf748b (127.0.98.3:46427)
I20260619 12:39:29.222935 19063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:41687
I20260619 12:39:29.226526 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:39:29.232051 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.228489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58362 (local address 127.0.98.62:44523)
0619 12:39:29.228618 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:29.228621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:29.228829 (+ 208us) server_negotiation.cc:407] Connection header received
0619 12:39:29.228899 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:29.228900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:29.228938 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:29.229005 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:29.229234 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.229738 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.230508 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.230707 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.230795 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:29.231679 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:29.231682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:29.231684 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:29.231694 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:29.231711 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:29.231718 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:29.231826 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:29.231909 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 12:39:29.231941 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:39:29.234743 19063 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:39:29.237710 19063 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:39:29.252091 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.246922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40925 (local address 127.0.0.1:52774)
0619 12:39:29.247567 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:29.247581 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:29.247653 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:29.248218 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:29.248220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:29.248235 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:29.248338 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:29.248344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.249323 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.249326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:29.250535 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:29.250544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.251535 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.251538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:29.251646 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:29.251687 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 12:39:29.251802 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 12:39:29.251925 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
I20260619 12:39:29.252115 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.247339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.0.98.1:40925)
0619 12:39:29.247637 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:29.247650 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:29.247677 (+ 27us) server_negotiation.cc:407] Connection header received
0619 12:39:29.247755 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:29.247760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:29.247958 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:29.248106 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:29.248466 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.249191 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.250664 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.251374 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.251506 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:29.251708 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:29.251832 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 12:39:29.251940 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":160,"threads_started":1}
I20260619 12:39:29.252677 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.246968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.0.98.2:35001)
0619 12:39:29.247349 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:29.247364 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:29.247625 (+ 261us) server_negotiation.cc:407] Connection header received
0619 12:39:29.247712 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:29.247719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:29.247902 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:29.248064 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:29.248324 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.249523 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.251010 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.251797 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.251894 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:29.252065 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:29.252424 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 12:39:29.252530 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":121,"threads_started":1}
I20260619 12:39:29.252137 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.246143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46427 (local address 127.0.0.1:34256)
0619 12:39:29.246544 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:29.246599 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:29.246726 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:29.248235 (+ 1509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:29.248239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:29.248257 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:29.248355 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:29.248363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.249821 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.249824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:29.250914 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:29.250924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.251823 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.251826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:29.251907 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:29.251928 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 12:39:29.251975 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 12:39:29.252031 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260619 12:39:29.253262 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.246324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.0.98.3:46427)
0619 12:39:29.247837 (+ 1513us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:29.247848 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:29.247867 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:39:29.247931 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:29.247936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:29.248090 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:29.248505 (+ 415us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:29.248641 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.249694 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.251040 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.251672 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.252767 (+ 1095us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:29.252957 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:29.253011 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 12:39:29.253105 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1399,"thread_start_us":147,"threads_started":1}
I20260619 12:39:29.253402 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.246544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35001 (local address 127.0.0.1:52440)
0619 12:39:29.247379 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:29.247393 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:29.247516 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:29.248053 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:29.248059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:29.248080 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:29.248222 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:29.248229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.249651 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.249654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:29.250880 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:29.250891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.252131 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.252137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:29.252245 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:29.252271 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 12:39:29.252392 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 12:39:29.252466 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260619 12:39:29.255558 19187 tablet_service.cc:1467] Processing CreateTablet for tablet 2338556c1c4c46aaa7df36e59ad237d7 (DEFAULT_TABLE table=test-table [id=241e13c777284545a4e58f648e1f725f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:29.256150 19318 tablet_service.cc:1467] Processing CreateTablet for tablet 2338556c1c4c46aaa7df36e59ad237d7 (DEFAULT_TABLE table=test-table [id=241e13c777284545a4e58f648e1f725f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:29.256680 19187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2338556c1c4c46aaa7df36e59ad237d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:29.257187 19318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2338556c1c4c46aaa7df36e59ad237d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:29.256860 19449 tablet_service.cc:1467] Processing CreateTablet for tablet 2338556c1c4c46aaa7df36e59ad237d7 (DEFAULT_TABLE table=test-table [id=241e13c777284545a4e58f648e1f725f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:29.257766 19449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2338556c1c4c46aaa7df36e59ad237d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:29.263007 19532 tablet_bootstrap.cc:492] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2: Bootstrap starting.
I20260619 12:39:29.264631 19533 tablet_bootstrap.cc:492] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62: Bootstrap starting.
I20260619 12:39:29.265825 19532 tablet_bootstrap.cc:654] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:29.266563 19533 tablet_bootstrap.cc:654] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:29.266546 19532 log.cc:824] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:29.267198 19533 log.cc:824] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:29.267776 19534 tablet_bootstrap.cc:492] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b: Bootstrap starting.
I20260619 12:39:29.268615 19532 tablet_bootstrap.cc:492] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2: No bootstrap required, opened a new log
I20260619 12:39:29.268829 19532 ts_tablet_manager.cc:1397] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:39:29.269148 19533 tablet_bootstrap.cc:492] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62: No bootstrap required, opened a new log
I20260619 12:39:29.269337 19533 ts_tablet_manager.cc:1397] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 12:39:29.269901 19534 tablet_bootstrap.cc:654] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:29.270694 19534 log.cc:824] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:29.272717 19534 tablet_bootstrap.cc:492] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b: No bootstrap required, opened a new log
I20260619 12:39:29.272912 19534 ts_tablet_manager.cc:1397] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:39:29.272805 19532 raft_consensus.cc:348] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868d9a8a39e44889b109ad0e8adf748b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46427 } } peers { permanent_uuid: "48474812788049fba7c40dfb143688e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40925 } } peers { permanent_uuid: "7057a7a252c44602935edf3196d79e62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } }
I20260619 12:39:29.273104 19532 raft_consensus.cc:374] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:29.273157 19532 raft_consensus.cc:729] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48474812788049fba7c40dfb143688e2, State: Initialized, Role: FOLLOWER
I20260619 12:39:29.273663 19532 consensus_queue.cc:260] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868d9a8a39e44889b109ad0e8adf748b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46427 } } peers { permanent_uuid: "48474812788049fba7c40dfb143688e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40925 } } peers { permanent_uuid: "7057a7a252c44602935edf3196d79e62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } }
I20260619 12:39:29.273711 19533 raft_consensus.cc:348] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868d9a8a39e44889b109ad0e8adf748b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46427 } } peers { permanent_uuid: "48474812788049fba7c40dfb143688e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40925 } } peers { permanent_uuid: "7057a7a252c44602935edf3196d79e62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } }
I20260619 12:39:29.273981 19533 raft_consensus.cc:374] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:29.274051 19533 raft_consensus.cc:729] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7057a7a252c44602935edf3196d79e62, State: Initialized, Role: FOLLOWER
I20260619 12:39:29.274420 19253 heartbeater.cc:503] Master 127.0.98.62:44523 was elected leader, sending a full tablet report...
I20260619 12:39:29.274501 19533 consensus_queue.cc:260] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868d9a8a39e44889b109ad0e8adf748b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46427 } } peers { permanent_uuid: "48474812788049fba7c40dfb143688e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40925 } } peers { permanent_uuid: "7057a7a252c44602935edf3196d79e62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } }
I20260619 12:39:29.275226 19384 heartbeater.cc:503] Master 127.0.98.62:44523 was elected leader, sending a full tablet report...
I20260619 12:39:29.275550 19533 ts_tablet_manager.cc:1428] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260619 12:39:29.276394 19532 ts_tablet_manager.cc:1428] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 12:39:29.277292 19534 raft_consensus.cc:348] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868d9a8a39e44889b109ad0e8adf748b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46427 } } peers { permanent_uuid: "48474812788049fba7c40dfb143688e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40925 } } peers { permanent_uuid: "7057a7a252c44602935edf3196d79e62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } }
I20260619 12:39:29.277519 19534 raft_consensus.cc:374] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:29.277576 19534 raft_consensus.cc:729] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868d9a8a39e44889b109ad0e8adf748b, State: Initialized, Role: FOLLOWER
I20260619 12:39:29.278017 19534 consensus_queue.cc:260] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868d9a8a39e44889b109ad0e8adf748b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46427 } } peers { permanent_uuid: "48474812788049fba7c40dfb143688e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40925 } } peers { permanent_uuid: "7057a7a252c44602935edf3196d79e62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } }
I20260619 12:39:29.278630 19515 heartbeater.cc:503] Master 127.0.98.62:44523 was elected leader, sending a full tablet report...
I20260619 12:39:29.279983 19534 ts_tablet_manager.cc:1428] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260619 12:39:29.307595 19254 tablet.cc:2369] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:39:29.348572 19538 raft_consensus.cc:482] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:39:29.348798 19538 raft_consensus.cc:504] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868d9a8a39e44889b109ad0e8adf748b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46427 } } peers { permanent_uuid: "48474812788049fba7c40dfb143688e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40925 } } peers { permanent_uuid: "7057a7a252c44602935edf3196d79e62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } }
I20260619 12:39:29.349963 19538 leader_election.cc:288] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 868d9a8a39e44889b109ad0e8adf748b (127.0.98.3:46427), 7057a7a252c44602935edf3196d79e62 (127.0.98.2:35001)
I20260619 12:39:29.354024 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.350330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47781 (local address 127.0.98.2:35001)
0619 12:39:29.350564 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:29.350570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:29.350822 (+ 252us) server_negotiation.cc:407] Connection header received
0619 12:39:29.350933 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:29.350936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:29.350984 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:29.351069 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:29.351364 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.351901 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.353028 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.353424 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.353507 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:29.353598 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:29.353803 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 12:39:29.353886 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260619 12:39:29.354029 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.350434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59595 (local address 127.0.98.3:46427)
0619 12:39:29.350640 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:29.350645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:29.350662 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:39:29.350755 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:29.350761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:29.350829 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:29.350903 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:29.351224 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.351871 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.353064 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.353445 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.353565 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:29.353624 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:29.353848 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 12:39:29.353921 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260619 12:39:29.354049 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.350213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35001 (local address 127.0.98.1:47781)
0619 12:39:29.350728 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:29.350741 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:29.350834 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:29.351089 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:29.351091 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:29.351105 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:29.351235 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:29.351241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.352034 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.352037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:29.352903 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:29.352912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.353555 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.353559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:29.353651 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:29.353701 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 12:39:29.353760 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 12:39:29.353856 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":169,"thread_start_us":89,"threads_started":1}
I20260619 12:39:29.354045 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.350213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46427 (local address 127.0.98.1:59595)
0619 12:39:29.350532 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:29.350548 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:29.350649 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:29.350930 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:29.350933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:29.350950 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:29.351105 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:29.351112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.351981 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.351986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:29.352909 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:29.352918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.353586 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.353590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:29.353700 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:29.353718 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 12:39:29.353820 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 12:39:29.353886 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":20,"thread_start_us":97,"threads_started":1}
I20260619 12:39:29.354709 19338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2338556c1c4c46aaa7df36e59ad237d7" candidate_uuid: "48474812788049fba7c40dfb143688e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7057a7a252c44602935edf3196d79e62" is_pre_election: true
I20260619 12:39:29.355007 19338 raft_consensus.cc:2454] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48474812788049fba7c40dfb143688e2 in term 0.
I20260619 12:39:29.355391 19469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2338556c1c4c46aaa7df36e59ad237d7" candidate_uuid: "48474812788049fba7c40dfb143688e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868d9a8a39e44889b109ad0e8adf748b" is_pre_election: true
I20260619 12:39:29.355582 19140 leader_election.cc:302] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48474812788049fba7c40dfb143688e2, 7057a7a252c44602935edf3196d79e62; no voters:
I20260619 12:39:29.355705 19469 raft_consensus.cc:2454] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48474812788049fba7c40dfb143688e2 in term 0.
I20260619 12:39:29.355893 19538 raft_consensus.cc:2781] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:39:29.355993 19538 raft_consensus.cc:482] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:39:29.356051 19538 raft_consensus.cc:3037] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:29.357175 19538 raft_consensus.cc:504] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868d9a8a39e44889b109ad0e8adf748b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46427 } } peers { permanent_uuid: "48474812788049fba7c40dfb143688e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40925 } } peers { permanent_uuid: "7057a7a252c44602935edf3196d79e62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } }
I20260619 12:39:29.357563 19538 leader_election.cc:288] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [CANDIDATE]: Term 1 election: Requested vote from peers 868d9a8a39e44889b109ad0e8adf748b (127.0.98.3:46427), 7057a7a252c44602935edf3196d79e62 (127.0.98.2:35001)
I20260619 12:39:29.357864 19469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2338556c1c4c46aaa7df36e59ad237d7" candidate_uuid: "48474812788049fba7c40dfb143688e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868d9a8a39e44889b109ad0e8adf748b"
I20260619 12:39:29.357932 19338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2338556c1c4c46aaa7df36e59ad237d7" candidate_uuid: "48474812788049fba7c40dfb143688e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7057a7a252c44602935edf3196d79e62"
I20260619 12:39:29.357998 19469 raft_consensus.cc:3037] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:29.358075 19338 raft_consensus.cc:3037] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:29.359099 19469 raft_consensus.cc:2454] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48474812788049fba7c40dfb143688e2 in term 1.
I20260619 12:39:29.359109 19338 raft_consensus.cc:2454] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48474812788049fba7c40dfb143688e2 in term 1.
I20260619 12:39:29.359494 19142 leader_election.cc:302] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48474812788049fba7c40dfb143688e2, 868d9a8a39e44889b109ad0e8adf748b; no voters:
I20260619 12:39:29.359702 19538 raft_consensus.cc:2781] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:29.359930 19538 raft_consensus.cc:686] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [term 1 LEADER]: Becoming Leader. State: Replica: 48474812788049fba7c40dfb143688e2, State: Running, Role: LEADER
I20260619 12:39:29.360261 19538 consensus_queue.cc:237] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868d9a8a39e44889b109ad0e8adf748b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46427 } } peers { permanent_uuid: "48474812788049fba7c40dfb143688e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40925 } } peers { permanent_uuid: "7057a7a252c44602935edf3196d79e62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } }
I20260619 12:39:29.363688 19064 catalog_manager.cc:5495] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48474812788049fba7c40dfb143688e2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "48474812788049fba7c40dfb143688e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868d9a8a39e44889b109ad0e8adf748b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48474812788049fba7c40dfb143688e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7057a7a252c44602935edf3196d79e62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } health_report { overall_health: UNKNOWN } } }
I20260619 12:39:29.380396 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.376674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.0.98.1:40925)
0619 12:39:29.376819 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:29.376822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:29.376837 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:39:29.376963 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:29.376966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:29.377040 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:29.377186 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:39:29.377464 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.378007 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.378963 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:29.379171 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:29.379249 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:29.379543 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:39:29.379954 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:39:29.380067 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:29.380238 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 12:39:29.380275 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:39:29.386617 19338 raft_consensus.cc:1261] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62 [term 1 FOLLOWER]: Refusing update from remote peer 48474812788049fba7c40dfb143688e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:39:29.386617 19469 raft_consensus.cc:1261] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b [term 1 FOLLOWER]: Refusing update from remote peer 48474812788049fba7c40dfb143688e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:39:29.387403 19543 consensus_queue.cc:1035] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7057a7a252c44602935edf3196d79e62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:39:29.387666 19538 consensus_queue.cc:1035] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868d9a8a39e44889b109ad0e8adf748b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:39:29.396399 19548 mvcc.cc:205] Tried to move back new op lower bound from 7298550863399821312 to 7298550863302492160. Current Snapshot: MvccSnapshot[applied={T|T < 7298550863399821312}]
I20260619 12:39:29.396485 19547 mvcc.cc:205] Tried to move back new op lower bound from 7298550863399821312 to 7298550863302492160. Current Snapshot: MvccSnapshot[applied={T|T < 7298550863399821312}]
I20260619 12:39:29.405344 19058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:39:29.405565 19058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:39:29.408299 19058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0eabd9bc4e5d40a8ab7004d974489102: Sending DeleteTablet for 3 replicas of tablet 2338556c1c4c46aaa7df36e59ad237d7
I20260619 12:39:29.409236 19318 tablet_service.cc:1514] Processing DeleteTablet for tablet 2338556c1c4c46aaa7df36e59ad237d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:29 UTC) from {username='slave'} at 127.0.0.1:52440
I20260619 12:39:29.409320 19449 tablet_service.cc:1514] Processing DeleteTablet for tablet 2338556c1c4c46aaa7df36e59ad237d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:29 UTC) from {username='slave'} at 127.0.0.1:34256
I20260619 12:39:29.409411 19187 tablet_service.cc:1514] Processing DeleteTablet for tablet 2338556c1c4c46aaa7df36e59ad237d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:29 UTC) from {username='slave'} at 127.0.0.1:52774
I20260619 12:39:29.409874 19558 tablet_replica.cc:330] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2: stopping tablet replica
I20260619 12:39:29.410063 19557 tablet_replica.cc:330] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b: stopping tablet replica
I20260619 12:39:29.410236 19558 raft_consensus.cc:2229] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:39:29.410377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 19125
I20260619 12:39:29.410682 19558 raft_consensus.cc:2258] T 2338556c1c4c46aaa7df36e59ad237d7 P 48474812788049fba7c40dfb143688e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:39:29.411448 19557 raft_consensus.cc:2229] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:39:29.411836 19556 tablet_replica.cc:330] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62: stopping tablet replica
I20260619 12:39:29.411875 19557 raft_consensus.cc:2258] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:39:29.412182 19556 raft_consensus.cc:2229] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:39:29.412504 19556 raft_consensus.cc:2258] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:39:29.414096 19557 ts_tablet_manager.cc:1905] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:39:29.414191 19556 ts_tablet_manager.cc:1905] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:39:29.416477 19557 ts_tablet_manager.cc:1918] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:39:29.416601 19557 log.cc:1196] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/2338556c1c4c46aaa7df36e59ad237d7
I20260619 12:39:29.416905 19557 ts_tablet_manager.cc:1939] T 2338556c1c4c46aaa7df36e59ad237d7 P 868d9a8a39e44889b109ad0e8adf748b: Deleting consensus metadata
I20260619 12:39:29.417668 19556 ts_tablet_manager.cc:1918] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:39:29.417778 19556 log.cc:1196] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/2338556c1c4c46aaa7df36e59ad237d7
I20260619 12:39:29.418104 19556 ts_tablet_manager.cc:1939] T 2338556c1c4c46aaa7df36e59ad237d7 P 7057a7a252c44602935edf3196d79e62: Deleting consensus metadata
I20260619 12:39:29.418218 19051 catalog_manager.cc:4841] TS 868d9a8a39e44889b109ad0e8adf748b (127.0.98.3:46427): tablet 2338556c1c4c46aaa7df36e59ad237d7 (table test-table [id=241e13c777284545a4e58f648e1f725f]) successfully deleted
W20260619 12:39:29.419173 19051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40925 (error 108)
I20260619 12:39:29.419425 19049 catalog_manager.cc:4841] TS 7057a7a252c44602935edf3196d79e62 (127.0.98.2:35001): tablet 2338556c1c4c46aaa7df36e59ad237d7 (table test-table [id=241e13c777284545a4e58f648e1f725f]) successfully deleted
I20260619 12:39:29.420046 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 19256
I20260619 12:39:29.420398 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:29.420040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40925 (local address 127.0.0.1:52788)
0619 12:39:29.420207 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:29.420290 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 12:39:29.420934 19051 catalog_manager.cc:4584] TS 48474812788049fba7c40dfb143688e2 (127.0.98.1:40925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2338556c1c4c46aaa7df36e59ad237d7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40925: connect: Connection refused (error 111)
I20260619 12:39:29.426455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 19387
I20260619 12:39:29.432117 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 19034
2026-06-19T12:39:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1096 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:29 dist-test-slave-kdrb krb5kdc[19562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:29 dist-test-slave-kdrb krb5kdc[19562](info): set up 2 sockets
Jun 19 12:39:29 dist-test-slave-kdrb krb5kdc[19562](info): commencing operation
krb5kdc: starting...
W20260619 12:39:30.475535 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:30 dist-test-slave-kdrb krb5kdc[19562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:39:30.603456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32981
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:30.719372 19575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:30.719758 19575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:30.719859 19575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:30.724009 19575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:30.724128 19575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:30.724162 19575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:30.724205 19575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:30.724238 19575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:30.729816 19575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:30.731065 19575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:30.732232 19575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:30.741555 19581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:30.741580 19580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:30.741580 19583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:30.741757 19575 server_base.cc:1034] running on GCE node
I20260619 12:39:30.742372 19575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:30.743701 19575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:30.744911 19575 hybrid_clock.cc:648] HybridClock initialized: now 1781872770744892 us; error 38 us; skew 500 ppm
Jun 19 12:39:30 dist-test-slave-kdrb krb5kdc[19562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:30.747952 19575 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:39:30.748966 19575 webserver.cc:460] Webserver started at http://127.0.98.62:45097/ using document root <none> and password file <none>
I20260619 12:39:30.749644 19575 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:30.749740 19575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:30.749943 19575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:30.752257 19575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "ef84228e180d457b9e9108822f4c48f6"
format_stamp: "Formatted at 2026-06-19 12:39:30 on dist-test-slave-kdrb"
server_key: "ef4f4d284a1fff3793bdcfdd0670726c"
server_key_iv: "9ccd51b7db582f6386dc936ab3a0e59a"
server_key_version: "encryptionkey@0"
I20260619 12:39:30.752882 19575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef84228e180d457b9e9108822f4c48f6"
format_stamp: "Formatted at 2026-06-19 12:39:30 on dist-test-slave-kdrb"
server_key: "ef4f4d284a1fff3793bdcfdd0670726c"
server_key_iv: "9ccd51b7db582f6386dc936ab3a0e59a"
server_key_version: "encryptionkey@0"
I20260619 12:39:30.756980 19575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:39:30.759610 19590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:30.761008 19575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:30.761152 19575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "ef84228e180d457b9e9108822f4c48f6"
format_stamp: "Formatted at 2026-06-19 12:39:30 on dist-test-slave-kdrb"
server_key: "ef4f4d284a1fff3793bdcfdd0670726c"
server_key_iv: "9ccd51b7db582f6386dc936ab3a0e59a"
server_key_version: "encryptionkey@0"
I20260619 12:39:30.761282 19575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:30.772575 19575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:30.776012 19575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:30.776242 19575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:30.783802 19575 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32981
I20260619 12:39:30.783828 19642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32981 every 8 connection(s)
I20260619 12:39:30.785029 19575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:30.788187 19643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:30.790580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 19575
I20260619 12:39:30.790730 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:30.791043 392 external_mini_cluster.cc:1428] Setting key c56567026035d51db997e5f72c5a5846
I20260619 12:39:30.795116 19643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6: Bootstrap starting.
I20260619 12:39:30.797636 19643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:30.798491 19643 log.cc:824] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:39:30 dist-test-slave-kdrb krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:39:30.801190 19643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6: No bootstrap required, opened a new log
I20260619 12:39:30.806587 19643 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef84228e180d457b9e9108822f4c48f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32981 } }
I20260619 12:39:30.806921 19643 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:30.806973 19643 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef84228e180d457b9e9108822f4c48f6, State: Initialized, Role: FOLLOWER
I20260619 12:39:30.807533 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:30.792720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49738 (local address 127.0.98.62:32981)
0619 12:39:30.793227 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:30.793243 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:30.793294 (+ 51us) server_negotiation.cc:407] Connection header received
0619 12:39:30.794161 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:30.794183 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:30.794568 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:30.794946 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:30.795841 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:30.796871 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:30.797782 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:30.798302 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:30.798491 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:30.801495 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:30.801502 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:30.801522 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:30.801558 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:30.803757 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:30.804388 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:30.804395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:30.804403 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:30.804573 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:30.804909 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:30.804913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:30.804914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:30.805537 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:30.805763 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:30.806065 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 12:39:30.806410 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":155,"threads_started":1}
I20260619 12:39:30.807544 19643 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef84228e180d457b9e9108822f4c48f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32981 } }
I20260619 12:39:30.807768 19643 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:30.807843 19643 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:30.807953 19643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:30.809226 19643 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef84228e180d457b9e9108822f4c48f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32981 } }
I20260619 12:39:30.809631 19643 leader_election.cc:302] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef84228e180d457b9e9108822f4c48f6; no voters:
I20260619 12:39:30.810024 19643 leader_election.cc:288] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:30.810146 19648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:30.810400 19648 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [term 1 LEADER]: Becoming Leader. State: Replica: ef84228e180d457b9e9108822f4c48f6, State: Running, Role: LEADER
I20260619 12:39:30.810812 19648 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef84228e180d457b9e9108822f4c48f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32981 } }
I20260619 12:39:30.811398 19643 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:30.812544 19650 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef84228e180d457b9e9108822f4c48f6. Latest consensus state: current_term: 1 leader_uuid: "ef84228e180d457b9e9108822f4c48f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef84228e180d457b9e9108822f4c48f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32981 } } }
I20260619 12:39:30.812692 19650 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:30.813033 19649 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef84228e180d457b9e9108822f4c48f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef84228e180d457b9e9108822f4c48f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32981 } } }
I20260619 12:39:30.813141 19657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:30.813201 19649 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:30.817057 19657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:30.823881 19657 catalog_manager.cc:1347] Generated new cluster ID: e9c42dcf7b5546a6a930200e276cf680
I20260619 12:39:30.824028 19657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:30.836431 19657 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:30.837879 19657 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:30.845727 19657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6: Generated new TSK 0
I20260619 12:39:30.846554 19657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:39:30.883742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32981
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:31.018376 19669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:31.018724 19669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:31.018814 19669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:31.023118 19669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:31.023231 19669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:31.023332 19669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:39:31.029466 19669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:31.030810 19669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:31.031980 19669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:31.042389 19677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:31.042371 19675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:31.042413 19674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:31.042443 19669 server_base.cc:1034] running on GCE node
I20260619 12:39:31.043151 19669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:31.043949 19669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:31.045171 19669 hybrid_clock.cc:648] HybridClock initialized: now 1781872771045133 us; error 61 us; skew 500 ppm
Jun 19 12:39:31 dist-test-slave-kdrb krb5kdc[19562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:31.048211 19669 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:39:31.049213 19669 webserver.cc:460] Webserver started at http://127.0.98.1:42915/ using document root <none> and password file <none>
I20260619 12:39:31.049870 19669 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:31.049973 19669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:31.050272 19669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:31.052596 19669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "1121ff2a4d5d49558f9d060104146f30"
format_stamp: "Formatted at 2026-06-19 12:39:31 on dist-test-slave-kdrb"
server_key: "7c0d38d144c0c0fc29bfceefabad22ef"
server_key_iv: "6a794427ccfbe66ae68177ea79821661"
server_key_version: "encryptionkey@0"
I20260619 12:39:31.053169 19669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1121ff2a4d5d49558f9d060104146f30"
format_stamp: "Formatted at 2026-06-19 12:39:31 on dist-test-slave-kdrb"
server_key: "7c0d38d144c0c0fc29bfceefabad22ef"
server_key_iv: "6a794427ccfbe66ae68177ea79821661"
server_key_version: "encryptionkey@0"
I20260619 12:39:31.057364 19669 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 12:39:31.060050 19684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:31.061414 19669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:31.061563 19669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "1121ff2a4d5d49558f9d060104146f30"
format_stamp: "Formatted at 2026-06-19 12:39:31 on dist-test-slave-kdrb"
server_key: "7c0d38d144c0c0fc29bfceefabad22ef"
server_key_iv: "6a794427ccfbe66ae68177ea79821661"
server_key_version: "encryptionkey@0"
I20260619 12:39:31.061687 19669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:31.080758 19669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:31.084174 19669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:31.084405 19669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:31.085111 19669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:31.086257 19669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:31.086349 19669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:31.086432 19669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:31.086489 19669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:31.096050 19669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38675
I20260619 12:39:31.096117 19797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38675 every 8 connection(s)
I20260619 12:39:31.097111 19669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:39:31.101665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 19669
I20260619 12:39:31.101812 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:39:31.102141 392 external_mini_cluster.cc:1428] Setting key 562712fb6eeaead60395e4c5818708c5
Jun 19 12:39:31 dist-test-slave-kdrb krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:39:31.113780 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.099075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38847 (local address 127.0.98.62:32981)
0619 12:39:31.099259 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:31.099264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:31.100545 (+ 1281us) server_negotiation.cc:407] Connection header received
0619 12:39:31.101522 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:31.101525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:31.101586 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:31.101670 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:31.103264 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.104020 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.105849 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.106257 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.106397 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.109597 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:31.109600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:31.109602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:31.109626 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:31.111285 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:31.112004 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:31.112009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:31.112011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:31.112111 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:31.112547 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:31.112550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:31.112551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:31.112743 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:31.112827 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:31.113507 (+ 680us) server_negotiation.cc:299] Negotiation successful
0619 12:39:31.113643 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 12:39:31.114841 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.099570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32981 (local address 127.0.98.1:38847)
0619 12:39:31.100378 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:31.100422 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:31.101252 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:31.101925 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:31.101936 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:31.102452 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:31.103115 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:31.103131 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.104163 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.104169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:31.105698 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:31.105708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.106448 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.106451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:31.106610 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.107299 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:39:31.107317 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:31.109380 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:31.111418 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:31.111427 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:31.111451 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:31.111883 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:31.112207 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:31.112213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:31.112216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:31.112442 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:31.112857 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:31.112862 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:31.113268 (+ 406us) client_negotiation.cc:769] Sending connection context
0619 12:39:31.113445 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 12:39:31.113736 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:39:31.117070 19798 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32981
I20260619 12:39:31.117507 19798 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:31.118196 19798 heartbeater.cc:511] Master 127.0.98.62:32981 requested a full tablet report, sending...
I20260619 12:39:31.120133 19607 ts_manager.cc:194] Registered new tserver with Master: 1121ff2a4d5d49558f9d060104146f30 (127.0.98.1:38675)
I20260619 12:39:31.121348 19607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:39:31.134480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32981
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:31.248919 19803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:31.249236 19803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:31.249318 19803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:31.253671 19803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:31.253779 19803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:31.253883 19803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:39:31.259747 19803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:31.260962 19803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:31.262059 19803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:31.273509 19811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:31.273856 19803 server_base.cc:1034] running on GCE node
W20260619 12:39:31.273524 19809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:31.273511 19808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:31.274431 19803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:31.275174 19803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:31.276378 19803 hybrid_clock.cc:648] HybridClock initialized: now 1781872771276332 us; error 64 us; skew 500 ppm
Jun 19 12:39:31 dist-test-slave-kdrb krb5kdc[19562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:31.279654 19803 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:39:31.280710 19803 webserver.cc:460] Webserver started at http://127.0.98.2:44367/ using document root <none> and password file <none>
I20260619 12:39:31.281296 19803 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:31.281358 19803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:31.281531 19803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:31.283814 19803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "06bbbeb7f824422c90cca770df7e6e20"
format_stamp: "Formatted at 2026-06-19 12:39:31 on dist-test-slave-kdrb"
server_key: "33990fd4f6f5f9741d96befbac03cd49"
server_key_iv: "3d70bd1754347b99fb3e9f452056adfe"
server_key_version: "encryptionkey@0"
I20260619 12:39:31.284417 19803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "06bbbeb7f824422c90cca770df7e6e20"
format_stamp: "Formatted at 2026-06-19 12:39:31 on dist-test-slave-kdrb"
server_key: "33990fd4f6f5f9741d96befbac03cd49"
server_key_iv: "3d70bd1754347b99fb3e9f452056adfe"
server_key_version: "encryptionkey@0"
I20260619 12:39:31.289052 19803 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:39:31.292040 19818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:31.293498 19803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:31.293673 19803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "06bbbeb7f824422c90cca770df7e6e20"
format_stamp: "Formatted at 2026-06-19 12:39:31 on dist-test-slave-kdrb"
server_key: "33990fd4f6f5f9741d96befbac03cd49"
server_key_iv: "3d70bd1754347b99fb3e9f452056adfe"
server_key_version: "encryptionkey@0"
I20260619 12:39:31.293821 19803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:31.318452 19803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:31.321805 19803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:31.322021 19803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:31.322633 19803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:31.323609 19803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:31.323678 19803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:31.323751 19803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:31.323800 19803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:31.333650 19803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45457
I20260619 12:39:31.333741 19931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45457 every 8 connection(s)
I20260619 12:39:31.334692 19803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:39:31.341830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 19803
I20260619 12:39:31.342022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:39:31.342330 392 external_mini_cluster.cc:1428] Setting key 19b325fedcdfd35e37bc94d18629e763
Jun 19 12:39:31 dist-test-slave-kdrb krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:39:31.351219 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.337324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43601 (local address 127.0.98.62:32981)
0619 12:39:31.337482 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:31.337486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:31.338363 (+ 877us) server_negotiation.cc:407] Connection header received
0619 12:39:31.339351 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:31.339354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:31.339414 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:31.339577 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:31.340967 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.341539 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.343057 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.343381 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.343517 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.346648 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:31.346655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:31.346660 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:31.346701 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:31.348428 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:31.349207 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:31.349210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:31.349211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:31.349295 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:31.349781 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:31.349784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:31.349785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:31.350002 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:31.350088 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:31.350959 (+ 871us) server_negotiation.cc:299] Negotiation successful
0619 12:39:31.351087 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 12:39:31.352628 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.337632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32981 (local address 127.0.98.2:43601)
0619 12:39:31.338183 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:31.338233 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:31.339109 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:31.339797 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:31.339805 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:31.340220 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:31.340791 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:31.340807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.341697 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.341701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:31.342930 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:31.342941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.343646 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.343654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:31.343882 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.344636 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:39:31.344655 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:31.346421 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:31.348575 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:31.348583 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:31.348607 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:31.349075 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:31.349402 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:31.349407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:31.349411 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:31.349657 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:31.350122 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:31.350131 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:31.350555 (+ 424us) client_negotiation.cc:769] Sending connection context
0619 12:39:31.350872 (+ 317us) client_negotiation.cc:240] Negotiation successful
0619 12:39:31.351314 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":142,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:39:31.354090 19932 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32981
I20260619 12:39:31.354394 19932 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:31.354923 19932 heartbeater.cc:511] Master 127.0.98.62:32981 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:39:31.356432 19607 ts_manager.cc:194] Registered new tserver with Master: 06bbbeb7f824422c90cca770df7e6e20 (127.0.98.2:45457)
I20260619 12:39:31.357324 19607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:39:31.375644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32981
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:31.492865 19937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:31.493177 19937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:31.493224 19937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:31.497514 19937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:31.497597 19937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:31.497663 19937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:39:31.503548 19937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:31.504700 19937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:31.505764 19937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:31.516311 19945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:31.516304 19942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:31.516304 19943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:31.516706 19937 server_base.cc:1034] running on GCE node
I20260619 12:39:31.517278 19937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:31.518004 19937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:31.519208 19937 hybrid_clock.cc:648] HybridClock initialized: now 1781872771519188 us; error 36 us; skew 500 ppm
Jun 19 12:39:31 dist-test-slave-kdrb krb5kdc[19562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:31.522473 19937 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:39:31.523855 19937 webserver.cc:460] Webserver started at http://127.0.98.3:34065/ using document root <none> and password file <none>
I20260619 12:39:31.524526 19937 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:31.524637 19937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:31.524905 19937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:31.527271 19937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "31691a60a4ae44c396aa9b31df51abdc"
format_stamp: "Formatted at 2026-06-19 12:39:31 on dist-test-slave-kdrb"
server_key: "27e5665771c509a4b94f01c314984ef3"
server_key_iv: "3dda8d9d718133527e8f2edc1fc03f8a"
server_key_version: "encryptionkey@0"
I20260619 12:39:31.528025 19937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "31691a60a4ae44c396aa9b31df51abdc"
format_stamp: "Formatted at 2026-06-19 12:39:31 on dist-test-slave-kdrb"
server_key: "27e5665771c509a4b94f01c314984ef3"
server_key_iv: "3dda8d9d718133527e8f2edc1fc03f8a"
server_key_version: "encryptionkey@0"
I20260619 12:39:31.532456 19937 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:39:31.535364 19952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:31.537057 19937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:31.537328 19937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "31691a60a4ae44c396aa9b31df51abdc"
format_stamp: "Formatted at 2026-06-19 12:39:31 on dist-test-slave-kdrb"
server_key: "27e5665771c509a4b94f01c314984ef3"
server_key_iv: "3dda8d9d718133527e8f2edc1fc03f8a"
server_key_version: "encryptionkey@0"
I20260619 12:39:31.537477 19937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:31.565346 19937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:31.569378 19937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:31.569619 19937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:31.570282 19937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:31.571154 19937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:31.571229 19937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:31.571331 19937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:31.571372 19937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:31.580578 19937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37303
I20260619 12:39:31.580597 20065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37303 every 8 connection(s)
I20260619 12:39:31.581566 19937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:39:31.583019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 19937
I20260619 12:39:31.583145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:39:31.583521 392 external_mini_cluster.cc:1428] Setting key 0dcf4c7d5bef238e93652be93eb264d9
Jun 19 12:39:31 dist-test-slave-kdrb krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:39:31.596020 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.583619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51569 (local address 127.0.98.62:32981)
0619 12:39:31.583754 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:31.583758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:31.584949 (+ 1191us) server_negotiation.cc:407] Connection header received
0619 12:39:31.586056 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:31.586059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:31.586106 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:31.586173 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:31.587382 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.587998 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.589128 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.589345 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.589425 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.592058 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:31.592062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:31.592064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:31.592097 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:31.593634 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:31.594276 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:31.594279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:31.594280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:31.594367 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:31.594727 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:31.594729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:31.594730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:31.594921 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:31.595011 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:31.595679 (+ 668us) server_negotiation.cc:299] Negotiation successful
0619 12:39:31.595834 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 12:39:31.596908 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.584105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32981 (local address 127.0.98.3:51569)
0619 12:39:31.584755 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:31.584831 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:31.585859 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:31.586348 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:31.586355 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:31.586757 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:31.587249 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:31.587263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.588111 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.588114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:31.589021 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:31.589030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.589476 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.589478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:31.589618 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.590216 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:39:31.590233 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:31.591841 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:31.593812 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:31.593818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:31.593834 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:31.594158 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:31.594465 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:39:31.594468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:39:31.594469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:39:31.594639 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:39:31.595051 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:31.595057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:31.595350 (+ 293us) client_negotiation.cc:769] Sending connection context
0619 12:39:31.595613 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 12:39:31.595924 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":174,"threads_started":1}
I20260619 12:39:31.598284 20066 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32981
I20260619 12:39:31.598569 20066 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:31.599153 20066 heartbeater.cc:511] Master 127.0.98.62:32981 requested a full tablet report, sending...
I20260619 12:39:31.600504 19607 ts_manager.cc:194] Registered new tserver with Master: 31691a60a4ae44c396aa9b31df51abdc (127.0.98.3:37303)
I20260619 12:39:31.601184 19607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51569
I20260619 12:39:31.610389 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:39:31.620471 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.612805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49746 (local address 127.0.98.62:32981)
0619 12:39:31.612937 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:31.612941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:31.613117 (+ 176us) server_negotiation.cc:407] Connection header received
0619 12:39:31.613212 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:31.613252 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:31.613319 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:31.613393 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:31.614129 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.614652 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.615629 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.615943 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.616029 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.617371 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:31.617374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:31.617376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:31.617400 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:31.618837 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:31.619245 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:31.619248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:31.619249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:31.619326 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:31.619583 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:31.619586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:31.619587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:31.619845 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:31.619960 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:31.620210 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 12:39:31.620326 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:39:31.623958 19607 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:39:31.627043 19607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:39:31.642994 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.636404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.0.98.2:45457)
0619 12:39:31.637087 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:31.637099 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:31.637440 (+ 341us) server_negotiation.cc:407] Connection header received
0619 12:39:31.637696 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:31.637703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:31.637930 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:31.638089 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:31.638410 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.639510 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.641269 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.642245 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.642364 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.642520 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:31.642709 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 12:39:31.642798 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":142,"threads_started":1}
I20260619 12:39:31.642980 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.636282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45457 (local address 127.0.0.1:36018)
0619 12:39:31.637288 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:31.637345 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:31.637514 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:31.638093 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:31.638099 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:31.638119 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:31.638277 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:31.638284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.639864 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.639868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:31.641139 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:31.641149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.642386 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.642393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:31.642511 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.642550 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 12:39:31.642683 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 12:39:31.642813 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":86,"threads_started":1}
I20260619 12:39:31.643015 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.637502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38675 (local address 127.0.0.1:46548)
0619 12:39:31.637864 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:31.637876 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:31.637959 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:31.638803 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:31.638807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:31.638826 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:31.638927 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:31.638933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.640293 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.640296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:31.641483 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:31.641492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.642667 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.642673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:31.642764 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.642783 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 12:39:31.642853 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 12:39:31.642913 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":140,"threads_started":1}
I20260619 12:39:31.643370 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.637919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.0.98.1:38675)
0619 12:39:31.638312 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:31.638320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:31.638344 (+ 24us) server_negotiation.cc:407] Connection header received
0619 12:39:31.638427 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:31.638431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:31.638644 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:31.638806 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:31.639053 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.640166 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.641605 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.642336 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.642446 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.642603 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:31.642911 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 12:39:31.643221 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":123,"threads_started":1}
I20260619 12:39:31.643947 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.636356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37303 (local address 127.0.0.1:33226)
0619 12:39:31.637290 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:31.637346 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:31.637479 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:31.638277 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:31.638279 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:31.638295 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:31.638410 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:31.638417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.639916 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.639919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:31.641325 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:31.641335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.643080 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.643085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:31.643607 (+ 522us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.643634 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 12:39:31.643739 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 12:39:31.643814 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":70,"thread_start_us":105,"threads_started":1}
I20260619 12:39:31.644043 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.636598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33226 (local address 127.0.98.3:37303)
0619 12:39:31.637042 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:31.637055 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:31.637626 (+ 571us) server_negotiation.cc:407] Connection header received
0619 12:39:31.637701 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:31.637706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:31.637930 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:31.638089 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:31.638542 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.639762 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.641449 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.642245 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.642364 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.642520 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:31.643776 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0619 12:39:31.643879 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":134,"threads_started":1}
I20260619 12:39:31.646620 19732 tablet_service.cc:1467] Processing CreateTablet for tablet a086415c30e64320a5e454b5670550fc (DEFAULT_TABLE table=test-table [id=7656db910f2e4c7c8c64421ce01ae282]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:31.646622 19866 tablet_service.cc:1467] Processing CreateTablet for tablet a086415c30e64320a5e454b5670550fc (DEFAULT_TABLE table=test-table [id=7656db910f2e4c7c8c64421ce01ae282]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:31.647004 20000 tablet_service.cc:1467] Processing CreateTablet for tablet a086415c30e64320a5e454b5670550fc (DEFAULT_TABLE table=test-table [id=7656db910f2e4c7c8c64421ce01ae282]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:31.647861 19732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a086415c30e64320a5e454b5670550fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:31.647861 19866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a086415c30e64320a5e454b5670550fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:31.648123 20000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a086415c30e64320a5e454b5670550fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:31.654716 20083 tablet_bootstrap.cc:492] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc: Bootstrap starting.
I20260619 12:39:31.655925 20084 tablet_bootstrap.cc:492] T a086415c30e64320a5e454b5670550fc P 1121ff2a4d5d49558f9d060104146f30: Bootstrap starting.
I20260619 12:39:31.657068 20083 tablet_bootstrap.cc:654] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:31.657806 20084 tablet_bootstrap.cc:654] T a086415c30e64320a5e454b5670550fc P 1121ff2a4d5d49558f9d060104146f30: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:31.658206 20083 log.cc:824] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:31.658597 20084 log.cc:824] T a086415c30e64320a5e454b5670550fc P 1121ff2a4d5d49558f9d060104146f30: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:31.658761 20085 tablet_bootstrap.cc:492] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20: Bootstrap starting.
I20260619 12:39:31.660521 20083 tablet_bootstrap.cc:492] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc: No bootstrap required, opened a new log
I20260619 12:39:31.660537 20084 tablet_bootstrap.cc:492] T a086415c30e64320a5e454b5670550fc P 1121ff2a4d5d49558f9d060104146f30: No bootstrap required, opened a new log
I20260619 12:39:31.660741 20083 ts_tablet_manager.cc:1397] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:39:31.660755 20084 ts_tablet_manager.cc:1397] T a086415c30e64320a5e454b5670550fc P 1121ff2a4d5d49558f9d060104146f30: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 12:39:31.661590 20085 tablet_bootstrap.cc:654] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:31.662477 20085 log.cc:824] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:31.664748 20083 raft_consensus.cc:348] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbbeb7f824422c90cca770df7e6e20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45457 } } peers { permanent_uuid: "31691a60a4ae44c396aa9b31df51abdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } } peers { permanent_uuid: "1121ff2a4d5d49558f9d060104146f30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38675 } }
I20260619 12:39:31.664752 20084 raft_consensus.cc:348] T a086415c30e64320a5e454b5670550fc P 1121ff2a4d5d49558f9d060104146f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbbeb7f824422c90cca770df7e6e20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45457 } } peers { permanent_uuid: "31691a60a4ae44c396aa9b31df51abdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } } peers { permanent_uuid: "1121ff2a4d5d49558f9d060104146f30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38675 } }
I20260619 12:39:31.665033 20085 tablet_bootstrap.cc:492] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20: No bootstrap required, opened a new log
I20260619 12:39:31.665063 20083 raft_consensus.cc:374] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:31.665093 20084 raft_consensus.cc:374] T a086415c30e64320a5e454b5670550fc P 1121ff2a4d5d49558f9d060104146f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:31.665150 20083 raft_consensus.cc:729] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31691a60a4ae44c396aa9b31df51abdc, State: Initialized, Role: FOLLOWER
I20260619 12:39:31.665174 20084 raft_consensus.cc:729] T a086415c30e64320a5e454b5670550fc P 1121ff2a4d5d49558f9d060104146f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1121ff2a4d5d49558f9d060104146f30, State: Initialized, Role: FOLLOWER
I20260619 12:39:31.665261 20085 ts_tablet_manager.cc:1397] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 12:39:31.665710 20083 consensus_queue.cc:260] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbbeb7f824422c90cca770df7e6e20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45457 } } peers { permanent_uuid: "31691a60a4ae44c396aa9b31df51abdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } } peers { permanent_uuid: "1121ff2a4d5d49558f9d060104146f30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38675 } }
I20260619 12:39:31.665716 20084 consensus_queue.cc:260] T a086415c30e64320a5e454b5670550fc P 1121ff2a4d5d49558f9d060104146f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbbeb7f824422c90cca770df7e6e20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45457 } } peers { permanent_uuid: "31691a60a4ae44c396aa9b31df51abdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } } peers { permanent_uuid: "1121ff2a4d5d49558f9d060104146f30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38675 } }
I20260619 12:39:31.666821 20084 ts_tablet_manager.cc:1428] T a086415c30e64320a5e454b5670550fc P 1121ff2a4d5d49558f9d060104146f30: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 12:39:31.666823 20083 ts_tablet_manager.cc:1428] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:39:31.667213 19798 heartbeater.cc:503] Master 127.0.98.62:32981 was elected leader, sending a full tablet report...
I20260619 12:39:31.667640 20066 heartbeater.cc:503] Master 127.0.98.62:32981 was elected leader, sending a full tablet report...
I20260619 12:39:31.669275 20085 raft_consensus.cc:348] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbbeb7f824422c90cca770df7e6e20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45457 } } peers { permanent_uuid: "31691a60a4ae44c396aa9b31df51abdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } } peers { permanent_uuid: "1121ff2a4d5d49558f9d060104146f30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38675 } }
I20260619 12:39:31.669608 20085 raft_consensus.cc:374] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:31.669703 20085 raft_consensus.cc:729] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06bbbeb7f824422c90cca770df7e6e20, State: Initialized, Role: FOLLOWER
I20260619 12:39:31.670264 20085 consensus_queue.cc:260] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbbeb7f824422c90cca770df7e6e20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45457 } } peers { permanent_uuid: "31691a60a4ae44c396aa9b31df51abdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } } peers { permanent_uuid: "1121ff2a4d5d49558f9d060104146f30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38675 } }
I20260619 12:39:31.671006 19932 heartbeater.cc:503] Master 127.0.98.62:32981 was elected leader, sending a full tablet report...
I20260619 12:39:31.671451 20085 ts_tablet_manager.cc:1428] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
W20260619 12:39:31.832592 20067 tablet.cc:2369] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:39:31.836017 19933 tablet.cc:2369] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:39:31.845314 20091 raft_consensus.cc:482] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:39:31.845543 20091 raft_consensus.cc:504] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbbeb7f824422c90cca770df7e6e20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45457 } } peers { permanent_uuid: "31691a60a4ae44c396aa9b31df51abdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } } peers { permanent_uuid: "1121ff2a4d5d49558f9d060104146f30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38675 } }
I20260619 12:39:31.846794 20091 leader_election.cc:288] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31691a60a4ae44c396aa9b31df51abdc (127.0.98.3:37303), 1121ff2a4d5d49558f9d060104146f30 (127.0.98.1:38675)
W20260619 12:39:31.848584 19799 tablet.cc:2369] T a086415c30e64320a5e454b5670550fc P 1121ff2a4d5d49558f9d060104146f30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:39:31.851034 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.846997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37303 (local address 127.0.98.2:38717)
0619 12:39:31.847141 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:31.847159 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:31.847269 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:31.847595 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:31.847598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:31.847617 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:31.847751 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:31.847757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.848489 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.848493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:31.849724 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:31.849735 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.850611 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.850614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:31.850702 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.850722 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 12:39:31.850784 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 12:39:31.850884 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 12:39:31.851027 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.847154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38717 (local address 127.0.98.3:37303)
0619 12:39:31.847300 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:31.847305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:31.847323 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:39:31.847390 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:31.847393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:31.847435 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:31.847577 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:31.847869 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.848368 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.849862 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.850492 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.850598 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.850667 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:31.850828 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 12:39:31.850907 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 12:39:31.851292 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.847146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38675 (local address 127.0.98.2:33465)
0619 12:39:31.847461 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:31.847500 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:31.847637 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:31.847891 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:31.847894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:31.847909 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:31.848032 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:31.848037 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.848874 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.848878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:31.849979 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:31.849986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.850883 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.850886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:31.850977 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.850998 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 12:39:31.851073 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 12:39:31.851145 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":77,"threads_started":1}
I20260619 12:39:31.851308 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.847335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33465 (local address 127.0.98.1:38675)
0619 12:39:31.847460 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:31.847464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:31.847593 (+ 129us) server_negotiation.cc:407] Connection header received
0619 12:39:31.847750 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:31.847753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:31.847791 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:31.847870 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:31.848141 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.848777 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.850092 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.850747 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.850867 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.850921 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:31.851102 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 12:39:31.851183 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:39:31.851980 20020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a086415c30e64320a5e454b5670550fc" candidate_uuid: "06bbbeb7f824422c90cca770df7e6e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31691a60a4ae44c396aa9b31df51abdc" is_pre_election: true
I20260619 12:39:31.852029 19752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a086415c30e64320a5e454b5670550fc" candidate_uuid: "06bbbeb7f824422c90cca770df7e6e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1121ff2a4d5d49558f9d060104146f30" is_pre_election: true
I20260619 12:39:31.852301 20020 raft_consensus.cc:2454] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06bbbeb7f824422c90cca770df7e6e20 in term 0.
I20260619 12:39:31.852310 19752 raft_consensus.cc:2454] T a086415c30e64320a5e454b5670550fc P 1121ff2a4d5d49558f9d060104146f30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06bbbeb7f824422c90cca770df7e6e20 in term 0.
I20260619 12:39:31.852854 19821 leader_election.cc:302] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06bbbeb7f824422c90cca770df7e6e20, 1121ff2a4d5d49558f9d060104146f30, 31691a60a4ae44c396aa9b31df51abdc; no voters:
I20260619 12:39:31.853112 20091 raft_consensus.cc:2781] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:39:31.853214 20091 raft_consensus.cc:482] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:39:31.853276 20091 raft_consensus.cc:3037] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:31.854302 20091 raft_consensus.cc:504] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbbeb7f824422c90cca770df7e6e20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45457 } } peers { permanent_uuid: "31691a60a4ae44c396aa9b31df51abdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } } peers { permanent_uuid: "1121ff2a4d5d49558f9d060104146f30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38675 } }
I20260619 12:39:31.854734 20091 leader_election.cc:288] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [CANDIDATE]: Term 1 election: Requested vote from peers 31691a60a4ae44c396aa9b31df51abdc (127.0.98.3:37303), 1121ff2a4d5d49558f9d060104146f30 (127.0.98.1:38675)
I20260619 12:39:31.855161 20020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a086415c30e64320a5e454b5670550fc" candidate_uuid: "06bbbeb7f824422c90cca770df7e6e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31691a60a4ae44c396aa9b31df51abdc"
I20260619 12:39:31.855289 20020 raft_consensus.cc:3037] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:31.855296 19752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a086415c30e64320a5e454b5670550fc" candidate_uuid: "06bbbeb7f824422c90cca770df7e6e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1121ff2a4d5d49558f9d060104146f30"
I20260619 12:39:31.855510 19752 raft_consensus.cc:3037] T a086415c30e64320a5e454b5670550fc P 1121ff2a4d5d49558f9d060104146f30 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:31.856441 20020 raft_consensus.cc:2454] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06bbbeb7f824422c90cca770df7e6e20 in term 1.
I20260619 12:39:31.856663 19752 raft_consensus.cc:2454] T a086415c30e64320a5e454b5670550fc P 1121ff2a4d5d49558f9d060104146f30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06bbbeb7f824422c90cca770df7e6e20 in term 1.
I20260619 12:39:31.856856 19819 leader_election.cc:302] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06bbbeb7f824422c90cca770df7e6e20, 31691a60a4ae44c396aa9b31df51abdc; no voters:
I20260619 12:39:31.857043 20091 raft_consensus.cc:2781] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:31.857323 20091 raft_consensus.cc:686] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [term 1 LEADER]: Becoming Leader. State: Replica: 06bbbeb7f824422c90cca770df7e6e20, State: Running, Role: LEADER
I20260619 12:39:31.857697 20091 consensus_queue.cc:237] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbbeb7f824422c90cca770df7e6e20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45457 } } peers { permanent_uuid: "31691a60a4ae44c396aa9b31df51abdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } } peers { permanent_uuid: "1121ff2a4d5d49558f9d060104146f30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38675 } }
I20260619 12:39:31.861178 19607 catalog_manager.cc:5495] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06bbbeb7f824422c90cca770df7e6e20 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "06bbbeb7f824422c90cca770df7e6e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbbeb7f824422c90cca770df7e6e20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31691a60a4ae44c396aa9b31df51abdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1121ff2a4d5d49558f9d060104146f30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38675 } health_report { overall_health: UNKNOWN } } }
I20260619 12:39:31.879587 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.875814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36026 (local address 127.0.98.2:45457)
0619 12:39:31.875962 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:31.875966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:31.875982 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:39:31.876135 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:31.876138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:31.876192 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:31.876326 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:39:31.876581 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.877093 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.878003 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:31.878243 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:31.878362 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:31.878683 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:39:31.879120 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:39:31.879271 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:31.879305 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 12:39:31.879362 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:39:31.885530 19752 raft_consensus.cc:1261] T a086415c30e64320a5e454b5670550fc P 1121ff2a4d5d49558f9d060104146f30 [term 1 FOLLOWER]: Refusing update from remote peer 06bbbeb7f824422c90cca770df7e6e20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:39:31.885530 20020 raft_consensus.cc:1261] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc [term 1 FOLLOWER]: Refusing update from remote peer 06bbbeb7f824422c90cca770df7e6e20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:39:31.886293 20093 consensus_queue.cc:1035] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31691a60a4ae44c396aa9b31df51abdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:39:31.886487 20091 consensus_queue.cc:1035] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1121ff2a4d5d49558f9d060104146f30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:39:31.904438 19605 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:39:31.904608 19605 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:39:31.907002 19605 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef84228e180d457b9e9108822f4c48f6: Sending DeleteTablet for 3 replicas of tablet a086415c30e64320a5e454b5670550fc
I20260619 12:39:31.907825 19866 tablet_service.cc:1514] Processing DeleteTablet for tablet a086415c30e64320a5e454b5670550fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36018
I20260619 12:39:31.907846 20000 tablet_service.cc:1514] Processing DeleteTablet for tablet a086415c30e64320a5e454b5670550fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33226
I20260619 12:39:31.908394 20106 tablet_replica.cc:330] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20: stopping tablet replica
I20260619 12:39:31.908394 20107 tablet_replica.cc:330] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc: stopping tablet replica
I20260619 12:39:31.908398 19732 tablet_service.cc:1514] Processing DeleteTablet for tablet a086415c30e64320a5e454b5670550fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46548
I20260619 12:39:31.908716 20106 raft_consensus.cc:2229] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:39:31.908716 20107 raft_consensus.cc:2229] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:39:31.909025 20107 raft_consensus.cc:2258] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:39:31.909096 20106 raft_consensus.cc:2258] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:39:31.909837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 19669
I20260619 12:39:31.911327 20107 ts_tablet_manager.cc:1905] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:39:31.911327 20106 ts_tablet_manager.cc:1905] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:39:31.914315 20106 ts_tablet_manager.cc:1918] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:39:31.914317 20107 ts_tablet_manager.cc:1918] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:39:31.914423 20106 log.cc:1196] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-1/wal/wals/a086415c30e64320a5e454b5670550fc
I20260619 12:39:31.914423 20107 log.cc:1196] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/a086415c30e64320a5e454b5670550fc
I20260619 12:39:31.914765 20107 ts_tablet_manager.cc:1939] T a086415c30e64320a5e454b5670550fc P 31691a60a4ae44c396aa9b31df51abdc: Deleting consensus metadata
I20260619 12:39:31.914765 20106 ts_tablet_manager.cc:1939] T a086415c30e64320a5e454b5670550fc P 06bbbeb7f824422c90cca770df7e6e20: Deleting consensus metadata
I20260619 12:39:31.916006 19591 catalog_manager.cc:4841] TS 31691a60a4ae44c396aa9b31df51abdc (127.0.98.3:37303): tablet a086415c30e64320a5e454b5670550fc (table test-table [id=7656db910f2e4c7c8c64421ce01ae282]) successfully deleted
I20260619 12:39:31.916006 19594 catalog_manager.cc:4841] TS 06bbbeb7f824422c90cca770df7e6e20 (127.0.98.2:45457): tablet a086415c30e64320a5e454b5670550fc (table test-table [id=7656db910f2e4c7c8c64421ce01ae282]) successfully deleted
W20260619 12:39:31.920953 19593 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38675 (error 108)
I20260619 12:39:31.921140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 19803
I20260619 12:39:31.922667 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:31.922335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38675 (local address 127.0.0.1:46564)
0619 12:39:31.922483 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:31.922562 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 12:39:31.922955 19593 catalog_manager.cc:4584] TS 1121ff2a4d5d49558f9d060104146f30 (127.0.98.1:38675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a086415c30e64320a5e454b5670550fc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38675: connect: Connection refused (error 111)
I20260619 12:39:31.927425 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 19937
I20260619 12:39:31.933082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 19575
2026-06-19T12:39:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2501 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T12:39:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:31Z Disabled control of system clock
I20260619 12:39:31.963881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37317
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:39:32.075894 20113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:32.076193 20113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:32.076231 20113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:32.080339 20113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:32.080400 20113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:32.080416 20113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:32.080435 20113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:32.080449 20113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:32.085373 20113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:32.086480 20113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:32.087834 20113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:32.096201 20121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:32.096169 20118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:32.096318 20113 server_base.cc:1034] running on GCE node
W20260619 12:39:32.096788 20119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:32.097508 20113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:32.098675 20113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:32.099858 20113 hybrid_clock.cc:648] HybridClock initialized: now 1781872772099828 us; error 49 us; skew 500 ppm
I20260619 12:39:32.101771 20113 webserver.cc:460] Webserver started at http://127.0.98.62:43517/ using document root <none> and password file <none>
I20260619 12:39:32.102384 20113 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:32.102483 20113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:32.102718 20113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:32.104894 20113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "fdf1f76a06594f349175fa12fb0fd472"
format_stamp: "Formatted at 2026-06-19 12:39:32 on dist-test-slave-kdrb"
server_key: "f9040736d895ed97202c3b6017498aa6"
server_key_iv: "38899efec3896727535703508bf0010f"
server_key_version: "encryptionkey@0"
I20260619 12:39:32.105485 20113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "fdf1f76a06594f349175fa12fb0fd472"
format_stamp: "Formatted at 2026-06-19 12:39:32 on dist-test-slave-kdrb"
server_key: "f9040736d895ed97202c3b6017498aa6"
server_key_iv: "38899efec3896727535703508bf0010f"
server_key_version: "encryptionkey@0"
I20260619 12:39:32.109277 20113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:39:32.111734 20127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:32.112949 20113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:39:32.113098 20113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "fdf1f76a06594f349175fa12fb0fd472"
format_stamp: "Formatted at 2026-06-19 12:39:32 on dist-test-slave-kdrb"
server_key: "f9040736d895ed97202c3b6017498aa6"
server_key_iv: "38899efec3896727535703508bf0010f"
server_key_version: "encryptionkey@0"
I20260619 12:39:32.113214 20113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:32.138978 20113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:32.139848 20113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:32.140056 20113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:32.147315 20179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37317 every 8 connection(s)
I20260619 12:39:32.147315 20113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37317
I20260619 12:39:32.148415 20113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:32.150395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 20113
I20260619 12:39:32.150528 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:32.150852 392 external_mini_cluster.cc:1428] Setting key d32e2d1cf2bfc7bd0a06114a3d63a08c
I20260619 12:39:32.152161 20180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:32.158519 20180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472: Bootstrap starting.
I20260619 12:39:32.160609 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:32.152389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46672 (local address 127.0.98.62:37317)
0619 12:39:32.153051 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:32.153068 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:32.153105 (+ 37us) server_negotiation.cc:407] Connection header received
0619 12:39:32.153790 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:32.153808 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:32.154223 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:32.154596 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:32.155090 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.156174 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.157094 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.157594 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.157765 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:32.158727 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:32.158735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:32.158753 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:32.159171 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:32.159227 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:32.159248 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:32.159367 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:32.159577 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 12:39:32.159854 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":145,"threads_started":1}
I20260619 12:39:32.161134 20180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:32.162024 20180 log.cc:824] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:32.164323 20180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472: No bootstrap required, opened a new log
I20260619 12:39:32.168171 20180 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf1f76a06594f349175fa12fb0fd472" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } }
I20260619 12:39:32.168551 20180 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:32.168609 20180 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf1f76a06594f349175fa12fb0fd472, State: Initialized, Role: FOLLOWER
I20260619 12:39:32.169064 20180 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf1f76a06594f349175fa12fb0fd472" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } }
I20260619 12:39:32.169209 20180 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:32.169262 20180 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:32.169335 20180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:32.170323 20180 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf1f76a06594f349175fa12fb0fd472" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } }
I20260619 12:39:32.170663 20180 leader_election.cc:302] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdf1f76a06594f349175fa12fb0fd472; no voters:
I20260619 12:39:32.170950 20180 leader_election.cc:288] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:32.171111 20185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:32.171360 20185 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [term 1 LEADER]: Becoming Leader. State: Replica: fdf1f76a06594f349175fa12fb0fd472, State: Running, Role: LEADER
I20260619 12:39:32.171742 20185 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf1f76a06594f349175fa12fb0fd472" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } }
I20260619 12:39:32.172353 20180 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:32.173341 20187 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdf1f76a06594f349175fa12fb0fd472. Latest consensus state: current_term: 1 leader_uuid: "fdf1f76a06594f349175fa12fb0fd472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf1f76a06594f349175fa12fb0fd472" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } } }
I20260619 12:39:32.173493 20187 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:32.173756 20186 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdf1f76a06594f349175fa12fb0fd472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf1f76a06594f349175fa12fb0fd472" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } } }
I20260619 12:39:32.173882 20186 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:32.174157 20194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:32.177618 20194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:32.183733 20194 catalog_manager.cc:1347] Generated new cluster ID: 3dcd2cf068a44974ba0c065ceb7da083
I20260619 12:39:32.183831 20194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:32.198604 20194 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:32.199990 20194 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:32.207312 20194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472: Generated new TSK 0
I20260619 12:39:32.208281 20194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:32.217429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37317
--builtin_ntp_servers=127.0.98.20:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:39:32.331382 20204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:32.331759 20204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:32.331804 20204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:32.336063 20204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:32.336127 20204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:32.336184 20204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:39:32.341264 20204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:32.342469 20204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:32.343998 20204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:32.353755 20212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:32.353824 20210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:32.353816 20209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:32.354128 20204 server_base.cc:1034] running on GCE node
I20260619 12:39:32.354573 20204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:32.355347 20204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:32.356521 20204 hybrid_clock.cc:648] HybridClock initialized: now 1781872772356493 us; error 43 us; skew 500 ppm
I20260619 12:39:32.358489 20204 webserver.cc:460] Webserver started at http://127.0.98.1:35259/ using document root <none> and password file <none>
I20260619 12:39:32.359107 20204 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:32.359170 20204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:32.359395 20204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:32.361555 20204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/data/instance:
uuid: "b85d8e9b86264030b1c3ad11c62c965b"
format_stamp: "Formatted at 2026-06-19 12:39:32 on dist-test-slave-kdrb"
server_key: "38ccb91faa9315610b45fb12ab754a63"
server_key_iv: "562ccaa0b72fe1fffbcaadd4840c062e"
server_key_version: "encryptionkey@0"
I20260619 12:39:32.362150 20204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b85d8e9b86264030b1c3ad11c62c965b"
format_stamp: "Formatted at 2026-06-19 12:39:32 on dist-test-slave-kdrb"
server_key: "38ccb91faa9315610b45fb12ab754a63"
server_key_iv: "562ccaa0b72fe1fffbcaadd4840c062e"
server_key_version: "encryptionkey@0"
I20260619 12:39:32.366070 20204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:39:32.368848 20218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:32.370273 20204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 12:39:32.370440 20204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/wal
uuid: "b85d8e9b86264030b1c3ad11c62c965b"
format_stamp: "Formatted at 2026-06-19 12:39:32 on dist-test-slave-kdrb"
server_key: "38ccb91faa9315610b45fb12ab754a63"
server_key_iv: "562ccaa0b72fe1fffbcaadd4840c062e"
server_key_version: "encryptionkey@0"
I20260619 12:39:32.370576 20204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:32.385653 20204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:32.386409 20204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:32.386627 20204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:32.387249 20204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:32.388273 20204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:32.388345 20204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:32.388419 20204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:32.388463 20204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:32.397802 20204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35013
I20260619 12:39:32.397817 20331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35013 every 8 connection(s)
I20260619 12:39:32.398816 20204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:39:32.404500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 20204
I20260619 12:39:32.404690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:39:32.405009 392 external_mini_cluster.cc:1428] Setting key 12e6933580b93f4b216fd138815f6049
I20260619 12:39:32.407259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37317
--builtin_ntp_servers=127.0.98.20:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:39:32.410115 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:32.401032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37303 (local address 127.0.98.62:37317)
0619 12:39:32.401181 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:32.401185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:32.402342 (+ 1157us) server_negotiation.cc:407] Connection header received
0619 12:39:32.403822 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:32.403826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:32.403891 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:32.403999 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:32.405061 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.405769 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.407594 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.407827 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.407917 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:32.409391 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:32.409398 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:32.409403 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:32.409420 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:32.409448 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:32.409458 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:32.409540 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:32.409799 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 12:39:32.409898 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:39:32.410987 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:32.401483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:37303)
0619 12:39:32.402175 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:32.402220 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:32.403588 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:32.404190 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:32.404198 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:32.404296 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:32.404914 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:32.404930 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.405924 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.405927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:32.407373 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:32.407390 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.408005 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.408012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:32.408271 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:32.409086 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:39:32.409088 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:32.409140 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:39:32.409214 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:32.409602 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:32.409612 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:32.409617 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:39:32.409749 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 12:39:32.410008 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":130,"threads_started":1}
I20260619 12:39:32.412408 20332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37317
I20260619 12:39:32.412700 20332 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:32.413296 20332 heartbeater.cc:511] Master 127.0.98.62:37317 requested a full tablet report, sending...
I20260619 12:39:32.414994 20144 ts_manager.cc:194] Registered new tserver with Master: b85d8e9b86264030b1c3ad11c62c965b (127.0.98.1:35013)
I20260619 12:39:32.416340 20144 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:37303
W20260619 12:39:32.522912 20335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:32.523267 20335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:32.523344 20335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:32.527621 20335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:32.527734 20335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:32.527840 20335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:39:32.532914 20335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:32.534133 20335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:32.535629 20335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:32.545768 20340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:32.545800 20341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:32.545768 20343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:32.546875 20335 server_base.cc:1034] running on GCE node
I20260619 12:39:32.547519 20335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:32.548293 20335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:32.549437 20335 hybrid_clock.cc:648] HybridClock initialized: now 1781872772549388 us; error 79 us; skew 500 ppm
I20260619 12:39:32.552085 20335 webserver.cc:460] Webserver started at http://127.0.98.2:44033/ using document root <none> and password file <none>
I20260619 12:39:32.552925 20335 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:32.553050 20335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:32.553361 20335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:32.556362 20335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/data/instance:
uuid: "80e4a3be56f04d6cb4236c955ec9b3d2"
format_stamp: "Formatted at 2026-06-19 12:39:32 on dist-test-slave-kdrb"
server_key: "8fa588091fd399a31def9128c90ceb7f"
server_key_iv: "41c96a0974ff26f932b5a0f921e372df"
server_key_version: "encryptionkey@0"
I20260619 12:39:32.557036 20335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/wal/instance:
uuid: "80e4a3be56f04d6cb4236c955ec9b3d2"
format_stamp: "Formatted at 2026-06-19 12:39:32 on dist-test-slave-kdrb"
server_key: "8fa588091fd399a31def9128c90ceb7f"
server_key_iv: "41c96a0974ff26f932b5a0f921e372df"
server_key_version: "encryptionkey@0"
I20260619 12:39:32.561424 20335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 12:39:32.564384 20349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:32.565847 20335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 12:39:32.566015 20335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/wal
uuid: "80e4a3be56f04d6cb4236c955ec9b3d2"
format_stamp: "Formatted at 2026-06-19 12:39:32 on dist-test-slave-kdrb"
server_key: "8fa588091fd399a31def9128c90ceb7f"
server_key_iv: "41c96a0974ff26f932b5a0f921e372df"
server_key_version: "encryptionkey@0"
I20260619 12:39:32.566157 20335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:32.585474 20335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:32.586442 20335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:32.586696 20335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:32.587378 20335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:32.588424 20335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:32.588503 20335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:32.588579 20335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:32.588622 20335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:32.597548 20335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44951
I20260619 12:39:32.597565 20462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44951 every 8 connection(s)
I20260619 12:39:32.598512 20335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:39:32.605518 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 20335
I20260619 12:39:32.605640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:39:32.606017 392 external_mini_cluster.cc:1428] Setting key a58fa22335f9b38937c5bb02e326c155
I20260619 12:39:32.608222 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:32.600383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38693 (local address 127.0.98.62:37317)
0619 12:39:32.600544 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:32.600549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:32.601713 (+ 1164us) server_negotiation.cc:407] Connection header received
0619 12:39:32.603033 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:32.603035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:32.603083 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:32.603161 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:32.604098 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.604623 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.605915 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.606229 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.606354 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:32.607687 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:32.607692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:32.607694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:32.607708 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:32.607731 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:32.607742 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:32.607821 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:32.607969 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 12:39:32.608045 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 12:39:32.608215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37317
--builtin_ntp_servers=127.0.98.20:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:39:32.609133 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:32.600862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:38693)
0619 12:39:32.601539 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:32.601583 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:32.602839 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:32.603342 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:32.603350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:32.603437 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:32.603958 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:32.603970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.604743 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.604745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:32.605789 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:32.605799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.606421 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.606428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:32.606656 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:32.607347 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:39:32.607349 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:32.607410 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:39:32.607543 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:32.607828 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:32.607835 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:32.607839 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:39:32.607938 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 12:39:32.608178 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
I20260619 12:39:32.610456 20463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37317
I20260619 12:39:32.610764 20463 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:32.611408 20463 heartbeater.cc:511] Master 127.0.98.62:37317 requested a full tablet report, sending...
I20260619 12:39:32.612659 20144 ts_manager.cc:194] Registered new tserver with Master: 80e4a3be56f04d6cb4236c955ec9b3d2 (127.0.98.2:44951)
I20260619 12:39:32.613567 20144 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:38693
W20260619 12:39:32.725558 20466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:32.725862 20466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:32.725906 20466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:32.730078 20466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:32.730165 20466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:32.730227 20466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:39:32.735332 20466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:32.736564 20466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:32.737980 20466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:32.747826 20471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:32.747917 20466 server_base.cc:1034] running on GCE node
W20260619 12:39:32.747938 20472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:32.747826 20474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:32.748582 20466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:32.749290 20466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:32.750465 20466 hybrid_clock.cc:648] HybridClock initialized: now 1781872772750446 us; error 31 us; skew 500 ppm
I20260619 12:39:32.752774 20466 webserver.cc:460] Webserver started at http://127.0.98.3:33683/ using document root <none> and password file <none>
I20260619 12:39:32.753435 20466 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:32.753517 20466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:32.753702 20466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:32.756031 20466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/data/instance:
uuid: "ca9f3d2474654c1bbdadc0164967f3b7"
format_stamp: "Formatted at 2026-06-19 12:39:32 on dist-test-slave-kdrb"
server_key: "ee15c5ed130a3fcd50f835fedc4d4e5d"
server_key_iv: "c4f58e2e0baae47fb52925a3db91eb29"
server_key_version: "encryptionkey@0"
I20260619 12:39:32.756582 20466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ca9f3d2474654c1bbdadc0164967f3b7"
format_stamp: "Formatted at 2026-06-19 12:39:32 on dist-test-slave-kdrb"
server_key: "ee15c5ed130a3fcd50f835fedc4d4e5d"
server_key_iv: "c4f58e2e0baae47fb52925a3db91eb29"
server_key_version: "encryptionkey@0"
I20260619 12:39:32.760525 20466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:39:32.763248 20480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:32.764696 20466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:32.764860 20466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/wal
uuid: "ca9f3d2474654c1bbdadc0164967f3b7"
format_stamp: "Formatted at 2026-06-19 12:39:32 on dist-test-slave-kdrb"
server_key: "ee15c5ed130a3fcd50f835fedc4d4e5d"
server_key_iv: "c4f58e2e0baae47fb52925a3db91eb29"
server_key_version: "encryptionkey@0"
I20260619 12:39:32.765026 20466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:32.795040 20466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:32.795921 20466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:32.796141 20466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:32.796787 20466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:39:32.797783 20466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:39:32.797857 20466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:32.797937 20466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:39:32.797981 20466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:32.807372 20466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41641
I20260619 12:39:32.807390 20593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41641 every 8 connection(s)
I20260619 12:39:32.808393 20466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:39:32.815972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 20466
I20260619 12:39:32.816115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:39:32.816431 392 external_mini_cluster.cc:1428] Setting key c43fefc7392015e77ad21fd4f6676477
I20260619 12:39:32.818007 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:32.810234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44001 (local address 127.0.98.62:37317)
0619 12:39:32.810391 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:32.810396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:32.811621 (+ 1225us) server_negotiation.cc:407] Connection header received
0619 12:39:32.812998 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:32.813001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:32.813051 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:32.813130 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:32.814045 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.814580 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.815730 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.816000 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.816113 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:32.817481 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:32.817485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:32.817488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:32.817502 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:32.817547 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:32.817559 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:32.817625 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:32.817800 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 12:39:32.817867 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 12:39:32.819043 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:32.810729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:44001)
0619 12:39:32.811441 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:32.811518 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:32.812801 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:32.813301 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:32.813308 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:32.813395 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:39:32.813913 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:32.813927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.814687 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.814690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:32.815624 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:32.815632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.816173 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.816178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:32.816377 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:32.817145 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:39:32.817147 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:39:32.817206 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:39:32.817295 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:39:32.817627 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:39:32.817639 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:39:32.817647 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 12:39:32.817801 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 12:39:32.818115 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":146,"threads_started":1}
I20260619 12:39:32.820135 20594 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37317
I20260619 12:39:32.820399 20594 heartbeater.cc:464] Registering TS with master...
I20260619 12:39:32.820955 20594 heartbeater.cc:511] Master 127.0.98.62:37317 requested a full tablet report, sending...
I20260619 12:39:32.821992 20144 ts_manager.cc:194] Registered new tserver with Master: ca9f3d2474654c1bbdadc0164967f3b7 (127.0.98.3:41641)
I20260619 12:39:32.822577 20144 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44001
I20260619 12:39:32.830704 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:39:32.836645 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:32.832710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46680 (local address 127.0.98.62:37317)
0619 12:39:32.832878 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:32.832883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:32.833123 (+ 240us) server_negotiation.cc:407] Connection header received
0619 12:39:32.833190 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:32.833193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:32.833237 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:32.833313 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:32.833665 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.834196 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.835120 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.835337 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.835419 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:32.836190 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:32.836193 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:32.836195 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:32.836208 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:32.836229 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:32.836237 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:32.836302 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:32.836381 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 12:39:32.836432 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 12:39:32.840127 20144 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:39:32.843184 20144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:39:32.858425 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:32.853196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46026 (local address 127.0.98.2:44951)
0619 12:39:32.853645 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:32.853659 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:32.853684 (+ 25us) server_negotiation.cc:407] Connection header received
0619 12:39:32.853870 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:32.853874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:32.854032 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:32.854193 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:32.854645 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.855441 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.857022 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.857722 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.857843 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:32.858109 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:32.858162 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 12:39:32.858260 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":133,"threads_started":1}
I20260619 12:39:32.858420 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:32.852959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44951 (local address 127.0.0.1:46026)
0619 12:39:32.853596 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:32.853610 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:32.853701 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:32.854347 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:32.854350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:32.854365 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:32.854512 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:32.854520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.855591 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.855597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:32.856894 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:32.856904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.857849 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.857853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:32.857957 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:32.857994 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 12:39:32.858115 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 12:39:32.858249 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":10,"thread_start_us":85,"threads_started":1}
I20260619 12:39:32.858896 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:32.852117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35013 (local address 127.0.0.1:33572)
0619 12:39:32.852557 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:32.852621 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:32.852732 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:32.854173 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:32.854179 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:32.854203 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:32.854360 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:32.854367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.856172 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.856174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:32.857437 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:32.857447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.858495 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.858498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:32.858594 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:32.858617 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:39:32.858667 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 12:39:32.858744 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":66,"thread_start_us":106,"threads_started":1}
I20260619 12:39:32.859144 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:32.852406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41641 (local address 127.0.0.1:48038)
0619 12:39:32.853065 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:32.853079 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:32.853167 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:32.854366 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:32.854369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:32.854401 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:32.854597 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:32.854607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.856080 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.856085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:32.857400 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:32.857410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.858716 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.858719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:32.858811 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:32.858831 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 12:39:32.858909 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 12:39:32.858989 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":78,"spinlock_wait_cycles":4352,"thread_start_us":96,"threads_started":1}
I20260619 12:39:32.859222 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:32.853179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.0.98.3:41641)
0619 12:39:32.853927 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:32.853938 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:32.853958 (+ 20us) server_negotiation.cc:407] Connection header received
0619 12:39:32.854031 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:32.854036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:32.854198 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:32.854383 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:32.854737 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.855915 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.857527 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.858348 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.858469 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:32.858658 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:32.858968 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 12:39:32.859073 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":156,"threads_started":1}
I20260619 12:39:32.860399 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:32.852259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.0.98.1:35013)
0619 12:39:32.853608 (+ 1349us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:32.853628 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:32.853659 (+ 31us) server_negotiation.cc:407] Connection header received
0619 12:39:32.853744 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:32.853753 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:32.854000 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:32.854161 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:32.854506 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.856054 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.857577 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:32.858348 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:32.859836 (+ 1488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:32.860040 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:32.860094 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 12:39:32.860212 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1051,"thread_start_us":164,"threads_started":1}
I20260619 12:39:32.861841 20397 tablet_service.cc:1467] Processing CreateTablet for tablet fc8631bc70cc4895bca46586c5b5d639 (DEFAULT_TABLE table=test-table [id=028f71a4bf4e44d8986d7d6e72717eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:32.862838 20397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc8631bc70cc4895bca46586c5b5d639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:32.862623 20528 tablet_service.cc:1467] Processing CreateTablet for tablet fc8631bc70cc4895bca46586c5b5d639 (DEFAULT_TABLE table=test-table [id=028f71a4bf4e44d8986d7d6e72717eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:32.862985 20266 tablet_service.cc:1467] Processing CreateTablet for tablet fc8631bc70cc4895bca46586c5b5d639 (DEFAULT_TABLE table=test-table [id=028f71a4bf4e44d8986d7d6e72717eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:39:32.863718 20528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc8631bc70cc4895bca46586c5b5d639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:32.863955 20266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc8631bc70cc4895bca46586c5b5d639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:32.869932 20611 tablet_bootstrap.cc:492] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2: Bootstrap starting.
I20260619 12:39:32.871435 20613 tablet_bootstrap.cc:492] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7: Bootstrap starting.
I20260619 12:39:32.871435 20612 tablet_bootstrap.cc:492] T fc8631bc70cc4895bca46586c5b5d639 P b85d8e9b86264030b1c3ad11c62c965b: Bootstrap starting.
I20260619 12:39:32.872591 20611 tablet_bootstrap.cc:654] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:32.873476 20611 log.cc:824] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:32.873752 20613 tablet_bootstrap.cc:654] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:32.873808 20612 tablet_bootstrap.cc:654] T fc8631bc70cc4895bca46586c5b5d639 P b85d8e9b86264030b1c3ad11c62c965b: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:32.874646 20612 log.cc:824] T fc8631bc70cc4895bca46586c5b5d639 P b85d8e9b86264030b1c3ad11c62c965b: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:32.874646 20613 log.cc:824] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:32.876411 20611 tablet_bootstrap.cc:492] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2: No bootstrap required, opened a new log
I20260619 12:39:32.876639 20611 ts_tablet_manager.cc:1397] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 12:39:32.876757 20612 tablet_bootstrap.cc:492] T fc8631bc70cc4895bca46586c5b5d639 P b85d8e9b86264030b1c3ad11c62c965b: No bootstrap required, opened a new log
I20260619 12:39:32.876950 20612 ts_tablet_manager.cc:1397] T fc8631bc70cc4895bca46586c5b5d639 P b85d8e9b86264030b1c3ad11c62c965b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:39:32.876943 20613 tablet_bootstrap.cc:492] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7: No bootstrap required, opened a new log
I20260619 12:39:32.877115 20613 ts_tablet_manager.cc:1397] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:39:32.880354 20612 raft_consensus.cc:348] T fc8631bc70cc4895bca46586c5b5d639 P b85d8e9b86264030b1c3ad11c62c965b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b85d8e9b86264030b1c3ad11c62c965b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "80e4a3be56f04d6cb4236c955ec9b3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "ca9f3d2474654c1bbdadc0164967f3b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41641 } }
I20260619 12:39:32.880620 20612 raft_consensus.cc:374] T fc8631bc70cc4895bca46586c5b5d639 P b85d8e9b86264030b1c3ad11c62c965b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:32.880681 20612 raft_consensus.cc:729] T fc8631bc70cc4895bca46586c5b5d639 P b85d8e9b86264030b1c3ad11c62c965b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b85d8e9b86264030b1c3ad11c62c965b, State: Initialized, Role: FOLLOWER
I20260619 12:39:32.880776 20611 raft_consensus.cc:348] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b85d8e9b86264030b1c3ad11c62c965b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "80e4a3be56f04d6cb4236c955ec9b3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "ca9f3d2474654c1bbdadc0164967f3b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41641 } }
I20260619 12:39:32.881047 20611 raft_consensus.cc:374] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:32.881102 20611 raft_consensus.cc:729] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80e4a3be56f04d6cb4236c955ec9b3d2, State: Initialized, Role: FOLLOWER
I20260619 12:39:32.881199 20612 consensus_queue.cc:260] T fc8631bc70cc4895bca46586c5b5d639 P b85d8e9b86264030b1c3ad11c62c965b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b85d8e9b86264030b1c3ad11c62c965b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "80e4a3be56f04d6cb4236c955ec9b3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "ca9f3d2474654c1bbdadc0164967f3b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41641 } }
I20260619 12:39:32.881247 20613 raft_consensus.cc:348] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b85d8e9b86264030b1c3ad11c62c965b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "80e4a3be56f04d6cb4236c955ec9b3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "ca9f3d2474654c1bbdadc0164967f3b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41641 } }
I20260619 12:39:32.881487 20613 raft_consensus.cc:374] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:32.881542 20613 raft_consensus.cc:729] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9f3d2474654c1bbdadc0164967f3b7, State: Initialized, Role: FOLLOWER
I20260619 12:39:32.881536 20611 consensus_queue.cc:260] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b85d8e9b86264030b1c3ad11c62c965b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "80e4a3be56f04d6cb4236c955ec9b3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "ca9f3d2474654c1bbdadc0164967f3b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41641 } }
I20260619 12:39:32.881956 20613 consensus_queue.cc:260] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b85d8e9b86264030b1c3ad11c62c965b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "80e4a3be56f04d6cb4236c955ec9b3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "ca9f3d2474654c1bbdadc0164967f3b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41641 } }
I20260619 12:39:32.882290 20463 heartbeater.cc:503] Master 127.0.98.62:37317 was elected leader, sending a full tablet report...
I20260619 12:39:32.882346 20612 ts_tablet_manager.cc:1428] T fc8631bc70cc4895bca46586c5b5d639 P b85d8e9b86264030b1c3ad11c62c965b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 12:39:32.882519 20611 ts_tablet_manager.cc:1428] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:39:32.882658 20332 heartbeater.cc:503] Master 127.0.98.62:37317 was elected leader, sending a full tablet report...
I20260619 12:39:32.882803 20613 ts_tablet_manager.cc:1428] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:39:32.883625 20594 heartbeater.cc:503] Master 127.0.98.62:37317 was elected leader, sending a full tablet report...
W20260619 12:39:32.900161 20333 tablet.cc:2369] T fc8631bc70cc4895bca46586c5b5d639 P b85d8e9b86264030b1c3ad11c62c965b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:39:33.051682 20619 raft_consensus.cc:482] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:39:33.051894 20619 raft_consensus.cc:504] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b85d8e9b86264030b1c3ad11c62c965b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "80e4a3be56f04d6cb4236c955ec9b3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "ca9f3d2474654c1bbdadc0164967f3b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41641 } }
I20260619 12:39:33.053097 20619 leader_election.cc:288] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b85d8e9b86264030b1c3ad11c62c965b (127.0.98.1:35013), 80e4a3be56f04d6cb4236c955ec9b3d2 (127.0.98.2:44951)
I20260619 12:39:33.058395 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:33.053409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35013 (local address 127.0.98.3:50447)
0619 12:39:33.053545 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:33.053562 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:33.053663 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:33.054044 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:33.054048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:33.054069 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:33.054217 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:33.054227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:33.055267 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:33.055271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:33.056705 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:33.056710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:33.057825 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:33.057832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:33.057978 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:33.058012 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 12:39:33.058113 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 12:39:33.058206 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":69}
I20260619 12:39:33.058427 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:33.053897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49791 (local address 127.0.98.2:44951)
0619 12:39:33.054033 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:33.054037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:33.054194 (+ 157us) server_negotiation.cc:407] Connection header received
0619 12:39:33.054307 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:33.054310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:33.054364 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:33.054448 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:33.054680 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:33.055359 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:33.056876 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:33.057502 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:33.057655 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:33.057728 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:33.058112 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 12:39:33.058225 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:39:33.058410 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:33.053684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44951 (local address 127.0.98.3:49791)
0619 12:39:33.054094 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:33.054107 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:39:33.054202 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:39:33.054464 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:39:33.054466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:39:33.054479 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:39:33.054571 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:33.054575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:33.055452 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:33.055456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:33.056697 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:39:33.056710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:39:33.057714 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:39:33.057721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:39:33.057923 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:33.057969 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 12:39:33.058064 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 12:39:33.058171 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":39,"thread_start_us":178,"threads_started":1}
I20260619 12:39:33.059043 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:33.053530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50447 (local address 127.0.98.1:35013)
0619 12:39:33.053669 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:33.053675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:33.053694 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:39:33.053849 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:33.053854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:33.053916 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:33.054013 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:39:33.054337 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:33.055123 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:33.056889 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:33.057557 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:33.057666 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:33.057729 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:33.058819 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0619 12:39:33.058920 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:39:33.059268 20417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc8631bc70cc4895bca46586c5b5d639" candidate_uuid: "ca9f3d2474654c1bbdadc0164967f3b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80e4a3be56f04d6cb4236c955ec9b3d2" is_pre_election: true
W20260619 12:39:33.059347 20595 tablet.cc:2369] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:39:33.059643 20417 raft_consensus.cc:2454] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca9f3d2474654c1bbdadc0164967f3b7 in term 0.
I20260619 12:39:33.059713 20286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc8631bc70cc4895bca46586c5b5d639" candidate_uuid: "ca9f3d2474654c1bbdadc0164967f3b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b85d8e9b86264030b1c3ad11c62c965b" is_pre_election: true
I20260619 12:39:33.060034 20286 raft_consensus.cc:2454] T fc8631bc70cc4895bca46586c5b5d639 P b85d8e9b86264030b1c3ad11c62c965b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca9f3d2474654c1bbdadc0164967f3b7 in term 0.
I20260619 12:39:33.060253 20482 leader_election.cc:302] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80e4a3be56f04d6cb4236c955ec9b3d2, ca9f3d2474654c1bbdadc0164967f3b7; no voters:
I20260619 12:39:33.060554 20619 raft_consensus.cc:2781] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:39:33.060662 20619 raft_consensus.cc:482] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:39:33.060724 20619 raft_consensus.cc:3037] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:33.061803 20619 raft_consensus.cc:504] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b85d8e9b86264030b1c3ad11c62c965b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "80e4a3be56f04d6cb4236c955ec9b3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "ca9f3d2474654c1bbdadc0164967f3b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41641 } }
I20260619 12:39:33.062240 20619 leader_election.cc:288] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [CANDIDATE]: Term 1 election: Requested vote from peers b85d8e9b86264030b1c3ad11c62c965b (127.0.98.1:35013), 80e4a3be56f04d6cb4236c955ec9b3d2 (127.0.98.2:44951)
I20260619 12:39:33.062645 20286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc8631bc70cc4895bca46586c5b5d639" candidate_uuid: "ca9f3d2474654c1bbdadc0164967f3b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b85d8e9b86264030b1c3ad11c62c965b"
I20260619 12:39:33.062767 20417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc8631bc70cc4895bca46586c5b5d639" candidate_uuid: "ca9f3d2474654c1bbdadc0164967f3b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80e4a3be56f04d6cb4236c955ec9b3d2"
I20260619 12:39:33.062819 20286 raft_consensus.cc:3037] T fc8631bc70cc4895bca46586c5b5d639 P b85d8e9b86264030b1c3ad11c62c965b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:33.062932 20417 raft_consensus.cc:3037] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:33.064090 20286 raft_consensus.cc:2454] T fc8631bc70cc4895bca46586c5b5d639 P b85d8e9b86264030b1c3ad11c62c965b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca9f3d2474654c1bbdadc0164967f3b7 in term 1.
I20260619 12:39:33.064162 20417 raft_consensus.cc:2454] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca9f3d2474654c1bbdadc0164967f3b7 in term 1.
I20260619 12:39:33.064503 20482 leader_election.cc:302] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80e4a3be56f04d6cb4236c955ec9b3d2, ca9f3d2474654c1bbdadc0164967f3b7; no voters:
I20260619 12:39:33.064679 20619 raft_consensus.cc:2781] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:33.064918 20619 raft_consensus.cc:686] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [term 1 LEADER]: Becoming Leader. State: Replica: ca9f3d2474654c1bbdadc0164967f3b7, State: Running, Role: LEADER
I20260619 12:39:33.065250 20619 consensus_queue.cc:237] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b85d8e9b86264030b1c3ad11c62c965b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "80e4a3be56f04d6cb4236c955ec9b3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } } peers { permanent_uuid: "ca9f3d2474654c1bbdadc0164967f3b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41641 } }
I20260619 12:39:33.068663 20144 catalog_manager.cc:5495] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca9f3d2474654c1bbdadc0164967f3b7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ca9f3d2474654c1bbdadc0164967f3b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b85d8e9b86264030b1c3ad11c62c965b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80e4a3be56f04d6cb4236c955ec9b3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca9f3d2474654c1bbdadc0164967f3b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41641 } health_report { overall_health: HEALTHY } } }
I20260619 12:39:33.097119 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:33.093636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48050 (local address 127.0.98.3:41641)
0619 12:39:33.093798 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:33.093801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:33.093814 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:39:33.093894 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:33.093896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:33.093952 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:33.094049 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:39:33.094281 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:33.094789 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:33.095677 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:33.095892 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:33.095973 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:33.096304 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:39:33.096772 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:39:33.096876 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:33.096930 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 12:39:33.096980 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 12:39:33.099745 20464 tablet.cc:2369] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:39:33.103047 20417 raft_consensus.cc:1261] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2 [term 1 FOLLOWER]: Refusing update from remote peer ca9f3d2474654c1bbdadc0164967f3b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:39:33.103047 20286 raft_consensus.cc:1261] T fc8631bc70cc4895bca46586c5b5d639 P b85d8e9b86264030b1c3ad11c62c965b [term 1 FOLLOWER]: Refusing update from remote peer ca9f3d2474654c1bbdadc0164967f3b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:39:33.103852 20619 consensus_queue.cc:1035] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85d8e9b86264030b1c3ad11c62c965b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:39:33.104044 20621 consensus_queue.cc:1035] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80e4a3be56f04d6cb4236c955ec9b3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:39:33.120430 20144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:39:33.120595 20144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:39:33.123243 20144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdf1f76a06594f349175fa12fb0fd472: Sending DeleteTablet for 3 replicas of tablet fc8631bc70cc4895bca46586c5b5d639
I20260619 12:39:33.124918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 20204
I20260619 12:39:33.124722 20397 tablet_service.cc:1514] Processing DeleteTablet for tablet fc8631bc70cc4895bca46586c5b5d639 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:33 UTC) from {username='slave'} at 127.0.0.1:46026
I20260619 12:39:33.125993 20528 tablet_service.cc:1514] Processing DeleteTablet for tablet fc8631bc70cc4895bca46586c5b5d639 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:39:33 UTC) from {username='slave'} at 127.0.0.1:48038
I20260619 12:39:33.126343 20633 tablet_replica.cc:330] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2: stopping tablet replica
I20260619 12:39:33.126719 20633 raft_consensus.cc:2229] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:39:33.127095 20633 raft_consensus.cc:2258] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:39:33.132830 20633 ts_tablet_manager.cc:1905] T fc8631bc70cc4895bca46586c5b5d639 P 80e4a3be56f04d6cb4236c955ec9b3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 12:39:33.134222 20129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35013 (error 108)
I20260619 12:39:33.134867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 20335
I20260619 12:39:33.135685 20634 tablet_replica.cc:330] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7: stopping tablet replica
I20260619 12:39:33.136055 20634 raft_consensus.cc:2229] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:39:33.136446 20634 raft_consensus.cc:2258] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:39:33.136909 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:33.136486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35013 (local address 127.0.0.1:33586)
0619 12:39:33.136710 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:33.136793 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260619 12:39:33.137109 20129 catalog_manager.cc:4584] TS b85d8e9b86264030b1c3ad11c62c965b (127.0.98.1:35013): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc8631bc70cc4895bca46586c5b5d639: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35013: connect: Connection refused (error 111)
I20260619 12:39:33.138167 20634 ts_tablet_manager.cc:1905] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:39:33.141167 20634 ts_tablet_manager.cc:1918] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:39:33.141368 20634 log.cc:1196] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7: Deleting WAL directory at /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872516972220-392-0/minicluster-data/ts-2/wal/wals/fc8631bc70cc4895bca46586c5b5d639
I20260619 12:39:33.141770 20634 ts_tablet_manager.cc:1939] T fc8631bc70cc4895bca46586c5b5d639 P ca9f3d2474654c1bbdadc0164967f3b7: Deleting consensus metadata
I20260619 12:39:33.142047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 20466
I20260619 12:39:33.143815 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:33.142226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44951 (local address 127.0.0.1:46034)
0619 12:39:33.143604 (+ 1378us) negotiation.cc:107] Waiting for socket to connect
0619 12:39:33.143685 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1214}
I20260619 12:39:33.148378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 20113
2026-06-19T12:39:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1214 ms)
[----------] 6 tests from EncryptionPolicyTest (10837 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:33 dist-test-slave-kdrb krb5kdc[20638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:33 dist-test-slave-kdrb krb5kdc[20638](info): set up 2 sockets
Jun 19 12:39:33 dist-test-slave-kdrb krb5kdc[20638](info): commencing operation
krb5kdc: starting...
W20260619 12:39:34.187760 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:34 dist-test-slave-kdrb krb5kdc[20638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:39:34.311614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:34.424499 20651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:34.424810 20651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:34.424855 20651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:34.428902 20651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:34.428966 20651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:34.428985 20651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:34.429004 20651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:34.429020 20651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:34.434427 20651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:34.435526 20651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:34.436522 20651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:34.445313 20659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:34.445348 20656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:34.445508 20651 server_base.cc:1034] running on GCE node
W20260619 12:39:34.445313 20657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:34.446272 20651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:34.447572 20651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:34.448774 20651 hybrid_clock.cc:648] HybridClock initialized: now 1781872774448763 us; error 33 us; skew 500 ppm
Jun 19 12:39:34 dist-test-slave-kdrb krb5kdc[20638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:34.456552 20651 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:39:34.457779 20651 webserver.cc:460] Webserver started at http://127.0.0.1:44643/ using document root <none> and password file <none>
I20260619 12:39:34.458420 20651 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:34.458489 20651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:34.458738 20651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:34.461269 20651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "54c5d26edbd64e889d5cd345d2cfa37b"
format_stamp: "Formatted at 2026-06-19 12:39:34 on dist-test-slave-kdrb"
server_key: "f1196991d6bc1f534fb849625b541479"
server_key_iv: "ab269f50efe2ec3c904cfbca80e2fe49"
server_key_version: "encryptionkey@0"
I20260619 12:39:34.461968 20651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "54c5d26edbd64e889d5cd345d2cfa37b"
format_stamp: "Formatted at 2026-06-19 12:39:34 on dist-test-slave-kdrb"
server_key: "f1196991d6bc1f534fb849625b541479"
server_key_iv: "ab269f50efe2ec3c904cfbca80e2fe49"
server_key_version: "encryptionkey@0"
I20260619 12:39:34.466135 20651 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:39:34.468811 20666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:34.470221 20651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 12:39:34.470382 20651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "54c5d26edbd64e889d5cd345d2cfa37b"
format_stamp: "Formatted at 2026-06-19 12:39:34 on dist-test-slave-kdrb"
server_key: "f1196991d6bc1f534fb849625b541479"
server_key_iv: "ab269f50efe2ec3c904cfbca80e2fe49"
server_key_version: "encryptionkey@0"
I20260619 12:39:34.470532 20651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:34.497937 20651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:34.501065 20651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:34.501288 20651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:34.508526 20651 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45259
I20260619 12:39:34.508550 20718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45259 every 8 connection(s)
I20260619 12:39:34.509544 20651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:34.512382 20719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:34.518056 20719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b: Bootstrap starting.
I20260619 12:39:34.518548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 20651
I20260619 12:39:34.518692 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:34.518970 392 external_mini_cluster.cc:1428] Setting key db3343bbfc96357965926348717e3e53
I20260619 12:39:34.520916 20719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:34.521831 20719 log.cc:824] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:34.524034 20719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b: No bootstrap required, opened a new log
I20260619 12:39:34.528115 20719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c5d26edbd64e889d5cd345d2cfa37b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45259 } }
I20260619 12:39:34.528415 20719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:34.528486 20719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c5d26edbd64e889d5cd345d2cfa37b, State: Initialized, Role: FOLLOWER
Jun 19 12:39:34 dist-test-slave-kdrb krb5kdc[20638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 12:39:34.529016 20719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c5d26edbd64e889d5cd345d2cfa37b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45259 } }
I20260619 12:39:34.529191 20719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:34.529269 20719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:34.529412 20719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:34.530647 20719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c5d26edbd64e889d5cd345d2cfa37b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45259 } }
I20260619 12:39:34.531018 20719 leader_election.cc:302] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54c5d26edbd64e889d5cd345d2cfa37b; no voters:
I20260619 12:39:34.531386 20719 leader_election.cc:288] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:34.531459 20724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:34.531726 20724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [term 1 LEADER]: Becoming Leader. State: Replica: 54c5d26edbd64e889d5cd345d2cfa37b, State: Running, Role: LEADER
I20260619 12:39:34.532033 20724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c5d26edbd64e889d5cd345d2cfa37b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45259 } }
I20260619 12:39:34.532829 20719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:34.533298 20726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54c5d26edbd64e889d5cd345d2cfa37b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c5d26edbd64e889d5cd345d2cfa37b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45259 } } }
I20260619 12:39:34.533418 20726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:34.533669 20725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54c5d26edbd64e889d5cd345d2cfa37b. Latest consensus state: current_term: 1 leader_uuid: "54c5d26edbd64e889d5cd345d2cfa37b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c5d26edbd64e889d5cd345d2cfa37b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45259 } } }
I20260619 12:39:34.533792 20725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:34.533825 20728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:34.535597 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:34.520441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39681 (local address 127.0.0.1:45259)
0619 12:39:34.520895 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:34.520915 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:34.520960 (+ 45us) server_negotiation.cc:407] Connection header received
0619 12:39:34.521863 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:34.521889 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:34.522341 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:34.522700 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:34.523384 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:34.524855 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:34.525696 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:34.526399 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:34.526579 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:34.529433 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:34.529446 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:34.529461 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:34.529514 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:34.531418 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:34.532121 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:34.532131 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:34.532139 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:34.532280 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:34.532661 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:34.532664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:34.532666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:34.533235 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:34.533484 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:34.534297 (+ 813us) server_negotiation.cc:299] Negotiation successful
0619 12:39:34.534536 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260619 12:39:34.537531 20728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:34.539232 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:34.543649 20728 catalog_manager.cc:1347] Generated new cluster ID: 1475490aa218484aa66cf21c418ea60c
I20260619 12:39:34.543717 20728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:34.552412 20728 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:34.553402 20728 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:34.558894 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:34.547862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44725 (local address 127.0.0.1:45259)
0619 12:39:34.547972 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:34.547977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:34.548820 (+ 843us) server_negotiation.cc:407] Connection header received
0619 12:39:34.549111 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:34.549115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:34.549190 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:34.549242 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:34.550583 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:34.551401 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:34.552661 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:34.553066 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:34.553142 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:34.555086 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:34.555090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:34.555093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:34.555121 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:34.556812 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:34.557400 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:34.557404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:34.557406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:34.557488 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:34.557864 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:34.557868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:34.557869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:34.558071 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:34.558182 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:34.558609 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 12:39:34.558717 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":40}
I20260619 12:39:34.566334 20728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54c5d26edbd64e889d5cd345d2cfa37b: Generated new TSK 0
I20260619 12:39:34.567402 20728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:34.574633 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 20651
W20260619 12:39:34.580379 20636 connection.cc:535] client connection to 127.0.0.1:45259 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45259): Connection reset by peer (error 104)
2026-06-19T12:39:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:34 dist-test-slave-kdrb krb5kdc[20750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:34 dist-test-slave-kdrb krb5kdc[20750](info): set up 2 sockets
Jun 19 12:39:34 dist-test-slave-kdrb krb5kdc[20750](info): commencing operation
krb5kdc: starting...
W20260619 12:39:35.611773 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:35 dist-test-slave-kdrb krb5kdc[20750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:39:35.739190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41407
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:35.853251 20763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:35.853631 20763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:35.853724 20763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:35.857930 20763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:35.858052 20763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:35.858090 20763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:35.858130 20763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:35.858166 20763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:35.864578 20763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41407
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:35.865854 20763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:35.866961 20763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:39:35.876318 20763 server_base.cc:1034] running on GCE node
W20260619 12:39:35.876286 20769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:35.876273 20771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:35.876277 20768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:35.877216 20763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:35.878556 20763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:35.879740 20763 hybrid_clock.cc:648] HybridClock initialized: now 1781872775879727 us; error 30 us; skew 500 ppm
Jun 19 12:39:35 dist-test-slave-kdrb krb5kdc[20750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:35.882918 20763 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:39:35.883994 20763 webserver.cc:460] Webserver started at http://127.0.0.1:39051/ using document root <none> and password file <none>
I20260619 12:39:35.884632 20763 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:35.884734 20763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:35.884984 20763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:35.887240 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "83fd5436d1ef4e139259063906faf7ff"
format_stamp: "Formatted at 2026-06-19 12:39:35 on dist-test-slave-kdrb"
server_key: "2b776b831dc921ad5ae2909b5d7fe54b"
server_key_iv: "63d197dc7b94fbed3f487d13af9d8feb"
server_key_version: "encryptionkey@0"
I20260619 12:39:35.887907 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "83fd5436d1ef4e139259063906faf7ff"
format_stamp: "Formatted at 2026-06-19 12:39:35 on dist-test-slave-kdrb"
server_key: "2b776b831dc921ad5ae2909b5d7fe54b"
server_key_iv: "63d197dc7b94fbed3f487d13af9d8feb"
server_key_version: "encryptionkey@0"
I20260619 12:39:35.892421 20763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:39:35.895192 20778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:35.896638 20763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:35.896823 20763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "83fd5436d1ef4e139259063906faf7ff"
format_stamp: "Formatted at 2026-06-19 12:39:35 on dist-test-slave-kdrb"
server_key: "2b776b831dc921ad5ae2909b5d7fe54b"
server_key_iv: "63d197dc7b94fbed3f487d13af9d8feb"
server_key_version: "encryptionkey@0"
I20260619 12:39:35.896967 20763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:35.912384 20763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:35.915570 20763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:35.915802 20763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:35.923079 20763 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41407
I20260619 12:39:35.923179 20830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41407 every 8 connection(s)
I20260619 12:39:35.924168 20763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:35.926149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 20763
I20260619 12:39:35.926312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:35.926625 392 external_mini_cluster.cc:1428] Setting key 015d41a937e30b8770c8bab17755cf61
I20260619 12:39:35.928638 20831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:35.936124 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff: Bootstrap starting.
I20260619 12:39:35.936793 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:35.928239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46233 (local address 127.0.0.1:41407)
0619 12:39:35.928696 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:35.928713 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:35.928780 (+ 67us) server_negotiation.cc:407] Connection header received
0619 12:39:35.929499 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:35.929519 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:35.929904 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:35.930233 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:35.930602 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:35.931760 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:35.933082 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:35.933596 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:35.933742 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:35.934798 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:35.934803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:35.934818 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:35.935277 (+ 459us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:35.935339 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:35.935363 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:35.935539 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:35.935725 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 12:39:35.935906 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":132,"threads_started":1}
I20260619 12:39:35.938827 20831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:35.939733 20831 log.cc:824] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:35.940114 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:35.942083 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff: No bootstrap required, opened a new log
I20260619 12:39:35.946282 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:35.942132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52709 (local address 127.0.0.1:41407)
0619 12:39:35.942272 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:35.942277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:35.942457 (+ 180us) server_negotiation.cc:407] Connection header received
0619 12:39:35.942618 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:35.942622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:35.942671 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:35.942753 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:35.943049 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:35.943636 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:35.944493 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:35.944727 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:35.944836 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:35.945740 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:35.945751 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:35.945754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:35.945767 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:35.945794 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:35.945807 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:35.945902 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:35.946049 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 12:39:35.946091 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 12:39:35.946121 20831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fd5436d1ef4e139259063906faf7ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41407 } }
I20260619 12:39:35.946511 20831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:35.946592 20831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fd5436d1ef4e139259063906faf7ff, State: Initialized, Role: FOLLOWER
I20260619 12:39:35.947111 20831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fd5436d1ef4e139259063906faf7ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41407 } }
I20260619 12:39:35.947252 20831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:35.947342 20831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:35.947471 20831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:35.948526 20831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fd5436d1ef4e139259063906faf7ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41407 } }
I20260619 12:39:35.948873 20831 leader_election.cc:302] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83fd5436d1ef4e139259063906faf7ff; no voters:
I20260619 12:39:35.949167 20831 leader_election.cc:288] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:35.949332 20841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:35.949597 20841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [term 1 LEADER]: Becoming Leader. State: Replica: 83fd5436d1ef4e139259063906faf7ff, State: Running, Role: LEADER
I20260619 12:39:35.949920 20841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fd5436d1ef4e139259063906faf7ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41407 } }
I20260619 12:39:35.950603 20831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:35.951822 20843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83fd5436d1ef4e139259063906faf7ff. Latest consensus state: current_term: 1 leader_uuid: "83fd5436d1ef4e139259063906faf7ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fd5436d1ef4e139259063906faf7ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41407 } } }
I20260619 12:39:35.951967 20843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:35.952262 20842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83fd5436d1ef4e139259063906faf7ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fd5436d1ef4e139259063906faf7ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41407 } } }
I20260619 12:39:35.952371 20842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:35.952724 20850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:35.956173 20850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:35.962895 20850 catalog_manager.cc:1347] Generated new cluster ID: 3aaf6d6c2e1345a78ee9ada2b66f8954
I20260619 12:39:35.963001 20850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:35.977411 20850 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:35.978801 20850 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:35.984018 20850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83fd5436d1ef4e139259063906faf7ff: Generated new TSK 0
I20260619 12:39:35.984809 20850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:35.991616 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 20763
W20260619 12:39:35.997303 20748 connection.cc:535] client connection to 127.0.0.1:41407 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41407): Connection reset by peer (error 104)
2026-06-19T12:39:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:36 dist-test-slave-kdrb krb5kdc[20862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:36 dist-test-slave-kdrb krb5kdc[20862](info): set up 2 sockets
Jun 19 12:39:36 dist-test-slave-kdrb krb5kdc[20862](info): commencing operation
krb5kdc: starting...
W20260619 12:39:37.035869 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:37 dist-test-slave-kdrb krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:39:37.168416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40159
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:37.287147 20875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:37.287581 20875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:37.287707 20875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:37.292069 20875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:37.292197 20875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:37.292240 20875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:37.292301 20875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:37.292366 20875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:37.298388 20875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40159
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:37.299777 20875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:37.300953 20875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:37.310812 20880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:37.310806 20881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:37.310901 20875 server_base.cc:1034] running on GCE node
W20260619 12:39:37.310804 20883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:37.311722 20875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:37.313043 20875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:37.314273 20875 hybrid_clock.cc:648] HybridClock initialized: now 1781872777314257 us; error 33 us; skew 500 ppm
Jun 19 12:39:37 dist-test-slave-kdrb krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:37.317401 20875 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:39:37.318442 20875 webserver.cc:460] Webserver started at http://127.0.0.1:40033/ using document root <none> and password file <none>
I20260619 12:39:37.319015 20875 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:37.319078 20875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:37.319254 20875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:37.321491 20875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "c2eab9a5fa5a4b6f8f620c9dd893ad24"
format_stamp: "Formatted at 2026-06-19 12:39:37 on dist-test-slave-kdrb"
server_key: "94c9b8c1a4e6c8e9820935f9eff40aa4"
server_key_iv: "c38fbbdad5bb56a6ec544f7ed7e0a7c7"
server_key_version: "encryptionkey@0"
I20260619 12:39:37.322068 20875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "c2eab9a5fa5a4b6f8f620c9dd893ad24"
format_stamp: "Formatted at 2026-06-19 12:39:37 on dist-test-slave-kdrb"
server_key: "94c9b8c1a4e6c8e9820935f9eff40aa4"
server_key_iv: "c38fbbdad5bb56a6ec544f7ed7e0a7c7"
server_key_version: "encryptionkey@0"
I20260619 12:39:37.326226 20875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:39:37.329177 20890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:37.330638 20875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 12:39:37.330792 20875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "c2eab9a5fa5a4b6f8f620c9dd893ad24"
format_stamp: "Formatted at 2026-06-19 12:39:37 on dist-test-slave-kdrb"
server_key: "94c9b8c1a4e6c8e9820935f9eff40aa4"
server_key_iv: "c38fbbdad5bb56a6ec544f7ed7e0a7c7"
server_key_version: "encryptionkey@0"
I20260619 12:39:37.330914 20875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:37.336077 20875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:37.339759 20875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:37.339929 20875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:37.348193 20875 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40159
I20260619 12:39:37.348223 20942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40159 every 8 connection(s)
I20260619 12:39:37.349262 20875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:37.352372 20943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:37.355417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 20875
I20260619 12:39:37.355589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:37.355935 392 external_mini_cluster.cc:1428] Setting key bee392eb8ecce2c3a8231fd3c5de208e
I20260619 12:39:37.359247 20943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24: Bootstrap starting.
I20260619 12:39:37.361730 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:37.357540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53597 (local address 127.0.0.1:40159)
0619 12:39:37.358096 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:37.358112 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:37.358153 (+ 41us) server_negotiation.cc:407] Connection header received
0619 12:39:37.358828 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:37.358847 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:37.359173 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:37.359617 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:37.360472 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:37.360476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:37.360490 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:37.360500 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:37.360559 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:37.360581 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:37.360736 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:37.360809 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 12:39:37.360979 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":170,"threads_started":1}
I20260619 12:39:37.361956 20943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:37.362775 20943 log.cc:824] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:37.365063 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:37.365373 20943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24: No bootstrap required, opened a new log
I20260619 12:39:37.369562 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:37.367410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42885 (local address 127.0.0.1:40159)
0619 12:39:37.367583 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:37.367588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:37.367927 (+ 339us) server_negotiation.cc:407] Connection header received
0619 12:39:37.368060 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:37.368063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:37.368104 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:37.368201 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:37.369154 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:37.369159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:37.369162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:37.369175 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:37.369210 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:37.369224 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:37.369323 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:37.369388 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 12:39:37.369431 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 12:39:37.370056 20943 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eab9a5fa5a4b6f8f620c9dd893ad24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40159 } }
I20260619 12:39:37.370412 20943 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:37.370491 20943 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2eab9a5fa5a4b6f8f620c9dd893ad24, State: Initialized, Role: FOLLOWER
I20260619 12:39:37.370937 20943 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eab9a5fa5a4b6f8f620c9dd893ad24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40159 } }
I20260619 12:39:37.371069 20943 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:37.371111 20943 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:37.371188 20943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:37.372226 20943 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eab9a5fa5a4b6f8f620c9dd893ad24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40159 } }
I20260619 12:39:37.372584 20943 leader_election.cc:302] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2eab9a5fa5a4b6f8f620c9dd893ad24; no voters:
I20260619 12:39:37.372870 20943 leader_election.cc:288] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:37.373154 20953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:37.373421 20953 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [term 1 LEADER]: Becoming Leader. State: Replica: c2eab9a5fa5a4b6f8f620c9dd893ad24, State: Running, Role: LEADER
I20260619 12:39:37.373795 20953 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eab9a5fa5a4b6f8f620c9dd893ad24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40159 } }
I20260619 12:39:37.374272 20943 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:37.375665 20955 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2eab9a5fa5a4b6f8f620c9dd893ad24. Latest consensus state: current_term: 1 leader_uuid: "c2eab9a5fa5a4b6f8f620c9dd893ad24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eab9a5fa5a4b6f8f620c9dd893ad24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40159 } } }
I20260619 12:39:37.375823 20955 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:37.376255 20954 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2eab9a5fa5a4b6f8f620c9dd893ad24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eab9a5fa5a4b6f8f620c9dd893ad24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40159 } } }
I20260619 12:39:37.376398 20954 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24 [sys.catalog]: This master's current role is: LEADER
W20260619 12:39:37.380203 20968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:39:37.380335 20968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:39:37.380403 20963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:37.381176 20963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:37.387405 20963 catalog_manager.cc:1347] Generated new cluster ID: 211ede34c82b447bbd26a3ace3f801ce
I20260619 12:39:37.387528 20963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:37.404367 20963 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:37.404762 20963 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:37.412271 20963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2eab9a5fa5a4b6f8f620c9dd893ad24: Generated new TSK 0
I20260619 12:39:37.413196 20963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:37.417892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 20875
2026-06-19T12:39:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:37 dist-test-slave-kdrb krb5kdc[20974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:37 dist-test-slave-kdrb krb5kdc[20974](info): set up 2 sockets
Jun 19 12:39:37 dist-test-slave-kdrb krb5kdc[20974](info): commencing operation
krb5kdc: starting...
W20260619 12:39:38.471954 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:38 dist-test-slave-kdrb krb5kdc[20974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:39:38.601064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:38.719919 20987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:38.720232 20987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:38.720276 20987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:38.724515 20987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:38.724601 20987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:38.724622 20987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:38.724639 20987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:38.724655 20987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:38.730548 20987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:38.731750 20987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:38.732833 20987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:38.742107 20992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:38.742143 20993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:38.742107 20995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:38.742377 20987 server_base.cc:1034] running on GCE node
I20260619 12:39:38.743144 20987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:38.744498 20987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:38.745713 20987 hybrid_clock.cc:648] HybridClock initialized: now 1781872778745683 us; error 47 us; skew 500 ppm
Jun 19 12:39:38 dist-test-slave-kdrb krb5kdc[20974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:38.748667 20987 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:39:38.749676 20987 webserver.cc:460] Webserver started at http://127.0.0.1:44883/ using document root <none> and password file <none>
I20260619 12:39:38.750267 20987 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:38.750360 20987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:38.750598 20987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:38.752808 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "d57da19478414f469175fa70429b81b3"
format_stamp: "Formatted at 2026-06-19 12:39:38 on dist-test-slave-kdrb"
server_key: "71869d94f976e6abd4126b9e415291b5"
server_key_iv: "199e1056bacc5ffa644d3187266a235f"
server_key_version: "encryptionkey@0"
I20260619 12:39:38.753407 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "d57da19478414f469175fa70429b81b3"
format_stamp: "Formatted at 2026-06-19 12:39:38 on dist-test-slave-kdrb"
server_key: "71869d94f976e6abd4126b9e415291b5"
server_key_iv: "199e1056bacc5ffa644d3187266a235f"
server_key_version: "encryptionkey@0"
I20260619 12:39:38.757264 20987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:39:38.759866 21002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:38.761181 20987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:39:38.761334 20987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "d57da19478414f469175fa70429b81b3"
format_stamp: "Formatted at 2026-06-19 12:39:38 on dist-test-slave-kdrb"
server_key: "71869d94f976e6abd4126b9e415291b5"
server_key_iv: "199e1056bacc5ffa644d3187266a235f"
server_key_version: "encryptionkey@0"
I20260619 12:39:38.761457 20987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:38.776194 20987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:38.779263 20987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:38.779549 20987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:38.786751 20987 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39003
I20260619 12:39:38.786768 21054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39003 every 8 connection(s)
I20260619 12:39:38.787818 20987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:38.788219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 20987
I20260619 12:39:38.788371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:38.788674 392 external_mini_cluster.cc:1428] Setting key 5bacb7bed35ccc81fe3841b46b78bb9f
I20260619 12:39:38.790975 21055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:38.796736 21055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3: Bootstrap starting.
Jun 19 12:39:38 dist-test-slave-kdrb krb5kdc[20974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 12:39:38.799376 21055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:38.800128 21055 log.cc:824] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:38.802533 21055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3: No bootstrap required, opened a new log
I20260619 12:39:38.803532 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:38.790131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40491 (local address 127.0.0.1:39003)
0619 12:39:38.790577 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:38.790592 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:38.790627 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:39:38.791341 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:38.791360 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:38.791781 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:38.792121 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:38.792951 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:38.793968 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:38.794842 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:38.795323 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:38.795522 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:38.798300 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:38.798305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:38.798318 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:38.798347 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:38.800117 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:38.800610 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:38.800615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:38.800619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:38.800732 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:38.801037 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:38.801040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:38.801042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:38.801521 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:38.801745 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:38.801991 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 12:39:38.802291 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":146,"threads_started":1}
I20260619 12:39:38.806465 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:38.806265 21055 raft_consensus.cc:348] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57da19478414f469175fa70429b81b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39003 } }
I20260619 12:39:38.806556 21055 raft_consensus.cc:374] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:38.806640 21055 raft_consensus.cc:729] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57da19478414f469175fa70429b81b3, State: Initialized, Role: FOLLOWER
I20260619 12:39:38.807062 21055 consensus_queue.cc:260] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57da19478414f469175fa70429b81b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39003 } }
I20260619 12:39:38.807212 21055 raft_consensus.cc:388] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:38.807283 21055 raft_consensus.cc:482] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:38.807420 21055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:38.808504 21055 raft_consensus.cc:504] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57da19478414f469175fa70429b81b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39003 } }
I20260619 12:39:38.808939 21055 leader_election.cc:302] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d57da19478414f469175fa70429b81b3; no voters:
I20260619 12:39:38.809293 21055 leader_election.cc:288] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:38.809427 21065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:38.809628 21065 raft_consensus.cc:686] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [term 1 LEADER]: Becoming Leader. State: Replica: d57da19478414f469175fa70429b81b3, State: Running, Role: LEADER
I20260619 12:39:38.809954 21065 consensus_queue.cc:237] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57da19478414f469175fa70429b81b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39003 } }
I20260619 12:39:38.810658 21055 sys_catalog.cc:564] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:38.811215 21066 sys_catalog.cc:455] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d57da19478414f469175fa70429b81b3. Latest consensus state: current_term: 1 leader_uuid: "d57da19478414f469175fa70429b81b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57da19478414f469175fa70429b81b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39003 } } }
I20260619 12:39:38.811348 21066 sys_catalog.cc:458] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:38.811741 21067 sys_catalog.cc:455] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d57da19478414f469175fa70429b81b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57da19478414f469175fa70429b81b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39003 } } }
I20260619 12:39:38.811857 21067 sys_catalog.cc:458] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:38.812386 21070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:38.816319 21070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:38.818504 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:38.808696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57555 (local address 127.0.0.1:39003)
0619 12:39:38.808849 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:38.808853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:38.809066 (+ 213us) server_negotiation.cc:407] Connection header received
0619 12:39:38.809236 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:38.809242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:38.809290 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:38.809373 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:38.810196 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:38.810897 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:38.812166 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:38.812512 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:38.812619 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:38.814281 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:38.814286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:38.814288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:38.814317 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:38.816054 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:38.816684 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:38.816688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:38.816690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:38.816780 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:38.817108 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:38.817111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:38.817113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:38.817332 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:38.817450 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:38.818212 (+ 762us) server_negotiation.cc:299] Negotiation successful
0619 12:39:38.818342 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 12:39:38.822844 21070 catalog_manager.cc:1347] Generated new cluster ID: 1d28ec5b79024a9ab8141f099b212919
I20260619 12:39:38.822928 21070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:38.836628 21070 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:38.837815 21070 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:38.845100 21070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d57da19478414f469175fa70429b81b3: Generated new TSK 0
I20260619 12:39:38.845916 21070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:38.854550 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 20987
2026-06-19T12:39:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:38 dist-test-slave-kdrb krb5kdc[21086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:38 dist-test-slave-kdrb krb5kdc[21086](info): set up 2 sockets
Jun 19 12:39:38 dist-test-slave-kdrb krb5kdc[21086](info): commencing operation
krb5kdc: starting...
W20260619 12:39:39.903843 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:39 dist-test-slave-kdrb krb5kdc[21086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:39:40.030968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43749
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:40.144476 21099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:40.144811 21099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:40.144902 21099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:40.149066 21099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:40.149165 21099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:40.149202 21099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:40.149260 21099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:40.149300 21099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:40.156154 21099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:40.157315 21099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:40.158380 21099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:40.167105 21107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:40.167090 21105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:40.167090 21104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:40.167304 21099 server_base.cc:1034] running on GCE node
I20260619 12:39:40.168047 21099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:40.169193 21099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:40.170359 21099 hybrid_clock.cc:648] HybridClock initialized: now 1781872780170337 us; error 36 us; skew 500 ppm
Jun 19 12:39:40 dist-test-slave-kdrb krb5kdc[21086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:40.180718 21099 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:39:40.181833 21099 webserver.cc:460] Webserver started at http://127.0.0.1:45109/ using document root <none> and password file <none>
I20260619 12:39:40.182466 21099 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:40.182534 21099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:40.182761 21099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:40.185091 21099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "4ee54e8c298e41b1b70257d123075aea"
format_stamp: "Formatted at 2026-06-19 12:39:40 on dist-test-slave-kdrb"
server_key: "9f25020eaca171450e7c2c7e29166f94"
server_key_iv: "cd5644941997e16a0633b61cae1c9c4b"
server_key_version: "encryptionkey@0"
I20260619 12:39:40.185742 21099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ee54e8c298e41b1b70257d123075aea"
format_stamp: "Formatted at 2026-06-19 12:39:40 on dist-test-slave-kdrb"
server_key: "9f25020eaca171450e7c2c7e29166f94"
server_key_iv: "cd5644941997e16a0633b61cae1c9c4b"
server_key_version: "encryptionkey@0"
I20260619 12:39:40.189875 21099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:39:40.192577 21114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:40.193960 21099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:40.194116 21099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "4ee54e8c298e41b1b70257d123075aea"
format_stamp: "Formatted at 2026-06-19 12:39:40 on dist-test-slave-kdrb"
server_key: "9f25020eaca171450e7c2c7e29166f94"
server_key_iv: "cd5644941997e16a0633b61cae1c9c4b"
server_key_version: "encryptionkey@0"
I20260619 12:39:40.194248 21099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:40.207037 21099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:40.210194 21099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:40.210427 21099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:40.218269 21099 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43749
I20260619 12:39:40.218295 21166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43749 every 8 connection(s)
I20260619 12:39:40.219305 21099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:40.222206 21167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:40.228051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 21099
I20260619 12:39:40.228075 21167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea: Bootstrap starting.
I20260619 12:39:40.228209 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:40.228523 392 external_mini_cluster.cc:1428] Setting key b50f2824868b5b6f24560654033c45be
I20260619 12:39:40.230803 21167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:40.231655 21167 log.cc:824] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:40.233547 21167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea: No bootstrap required, opened a new log
I20260619 12:39:40.237362 21167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee54e8c298e41b1b70257d123075aea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43749 } }
I20260619 12:39:40.237627 21167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:40.237709 21167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee54e8c298e41b1b70257d123075aea, State: Initialized, Role: FOLLOWER
I20260619 12:39:40.237763 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:40.229958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46613 (local address 127.0.0.1:43749)
0619 12:39:40.230420 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:40.230435 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:40.230471 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:39:40.231181 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:40.231201 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:40.231629 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:40.232000 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:40.232436 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:40.233415 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:40.234363 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:40.234845 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:40.235022 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:40.235952 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:40.235958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:40.235971 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:40.236376 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:40.236433 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:40.236456 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:40.236571 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:40.236717 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 12:39:40.236943 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":143,"threads_started":1}
I20260619 12:39:40.238271 21167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee54e8c298e41b1b70257d123075aea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43749 } }
I20260619 12:39:40.238437 21167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:40.238500 21167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:40.238593 21167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:40.239657 21167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee54e8c298e41b1b70257d123075aea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43749 } }
I20260619 12:39:40.240037 21167 leader_election.cc:302] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ee54e8c298e41b1b70257d123075aea; no voters:
I20260619 12:39:40.240404 21167 leader_election.cc:288] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:40.240523 21172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:40.240777 21172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [term 1 LEADER]: Becoming Leader. State: Replica: 4ee54e8c298e41b1b70257d123075aea, State: Running, Role: LEADER
I20260619 12:39:40.240845 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:40.241132 21172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee54e8c298e41b1b70257d123075aea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43749 } }
I20260619 12:39:40.241740 21167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:40.242143 21172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ee54e8c298e41b1b70257d123075aea. Latest consensus state: current_term: 1 leader_uuid: "4ee54e8c298e41b1b70257d123075aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee54e8c298e41b1b70257d123075aea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43749 } } }
I20260619 12:39:40.242398 21172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:40.242885 21180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:40.244436 21173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ee54e8c298e41b1b70257d123075aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee54e8c298e41b1b70257d123075aea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43749 } } }
I20260619 12:39:40.244544 21173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:40.246070 21180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:40.251394 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:40.243384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50445 (local address 127.0.0.1:43749)
0619 12:39:40.243513 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:40.243517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:40.246934 (+ 3417us) server_negotiation.cc:407] Connection header received
0619 12:39:40.247098 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:40.247101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:40.247156 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:40.247237 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:40.247584 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:40.248389 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:40.249501 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:40.249891 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:40.250008 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:40.250933 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:40.250938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:40.250941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:40.250954 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:40.250987 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:40.251001 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:40.251083 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:40.251188 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 12:39:40.251244 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 12:39:40.252254 21180 catalog_manager.cc:1347] Generated new cluster ID: f0b27957021d4586a0c34991c3dc0669
I20260619 12:39:40.252313 21180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:40.261760 21180 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:40.262682 21180 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:40.272343 21180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ee54e8c298e41b1b70257d123075aea: Generated new TSK 0
I20260619 12:39:40.273075 21180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:40.279534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 21099
2026-06-19T12:39:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:40 dist-test-slave-kdrb krb5kdc[21198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:40 dist-test-slave-kdrb krb5kdc[21198](info): set up 2 sockets
Jun 19 12:39:40 dist-test-slave-kdrb krb5kdc[21198](info): commencing operation
krb5kdc: starting...
W20260619 12:39:41.319837 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:41 dist-test-slave-kdrb krb5kdc[21198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:39:41.446764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:41.561990 21211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:41.562310 21211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:41.562359 21211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:41.566459 21211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:41.566545 21211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:41.566568 21211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:41.566594 21211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:41.566612 21211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:41.572053 21211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:41.573241 21211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:41.574322 21211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:41.583549 21217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:41.583582 21211 server_base.cc:1034] running on GCE node
W20260619 12:39:41.583712 21216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:41.583549 21219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:41.584350 21211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:41.585525 21211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:41.586696 21211 hybrid_clock.cc:648] HybridClock initialized: now 1781872781586676 us; error 39 us; skew 500 ppm
Jun 19 12:39:41 dist-test-slave-kdrb krb5kdc[21198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:41.589561 21211 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:39:41.590559 21211 webserver.cc:460] Webserver started at http://127.0.0.1:42693/ using document root <none> and password file <none>
I20260619 12:39:41.591148 21211 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:41.591243 21211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:41.591522 21211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:41.593664 21211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "75aa2b7b902c4d14ad0b97ba92fb33bc"
format_stamp: "Formatted at 2026-06-19 12:39:41 on dist-test-slave-kdrb"
server_key: "5d87a70cd0fea4c200d71670d6ef7ea1"
server_key_iv: "40d1958d62443ddfd46657531faf1313"
server_key_version: "encryptionkey@0"
I20260619 12:39:41.594281 21211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "75aa2b7b902c4d14ad0b97ba92fb33bc"
format_stamp: "Formatted at 2026-06-19 12:39:41 on dist-test-slave-kdrb"
server_key: "5d87a70cd0fea4c200d71670d6ef7ea1"
server_key_iv: "40d1958d62443ddfd46657531faf1313"
server_key_version: "encryptionkey@0"
I20260619 12:39:41.598227 21211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:39:41.600976 21226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:41.602240 21211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:39:41.602394 21211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "75aa2b7b902c4d14ad0b97ba92fb33bc"
format_stamp: "Formatted at 2026-06-19 12:39:41 on dist-test-slave-kdrb"
server_key: "5d87a70cd0fea4c200d71670d6ef7ea1"
server_key_iv: "40d1958d62443ddfd46657531faf1313"
server_key_version: "encryptionkey@0"
I20260619 12:39:41.602517 21211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:41.607208 21211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:41.610571 21211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:41.610786 21211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:41.618088 21211 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41197
I20260619 12:39:41.618101 21278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41197 every 8 connection(s)
I20260619 12:39:41.619074 21211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:41.621968 21279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:41.623148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 21211
I20260619 12:39:41.623312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:41.623662 392 external_mini_cluster.cc:1428] Setting key 77ad8d26fad48ee82afd3c5afcc5548b
I20260619 12:39:41.628432 21279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc: Bootstrap starting.
I20260619 12:39:41.629032 21282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:41.625126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39801 (local address 127.0.0.1:41197)
0619 12:39:41.625597 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:41.625612 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:41.625648 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:39:41.626298 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:41.626316 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:41.626604 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:41.626948 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:41.627790 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:41.627795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:41.627807 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:41.627817 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:41.627890 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:41.627911 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:41.628065 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:41.628135 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 12:39:41.628297 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":151,"threads_started":1}
I20260619 12:39:41.630942 21279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:41.631793 21279 log.cc:824] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:41.631956 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:41.633909 21279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc: No bootstrap required, opened a new log
I20260619 12:39:41.635931 21282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:41.634198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34255 (local address 127.0.0.1:41197)
0619 12:39:41.634328 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:41.634332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:41.634536 (+ 204us) server_negotiation.cc:407] Connection header received
0619 12:39:41.634654 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:41.634657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:41.634685 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:41.634749 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:41.635438 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:41.635441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:41.635443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:41.635452 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:41.635526 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:41.635548 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:41.635619 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:41.635722 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 12:39:41.635753 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:39:41.638273 21279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75aa2b7b902c4d14ad0b97ba92fb33bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } }
I20260619 12:39:41.638533 21279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:41.638594 21279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75aa2b7b902c4d14ad0b97ba92fb33bc, State: Initialized, Role: FOLLOWER
I20260619 12:39:41.639060 21279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75aa2b7b902c4d14ad0b97ba92fb33bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } }
I20260619 12:39:41.639221 21279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:41.639312 21279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:41.639417 21279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:41.640393 21279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75aa2b7b902c4d14ad0b97ba92fb33bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } }
I20260619 12:39:41.640743 21279 leader_election.cc:302] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75aa2b7b902c4d14ad0b97ba92fb33bc; no voters:
I20260619 12:39:41.641100 21279 leader_election.cc:288] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:41.641211 21289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:41.641470 21289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [term 1 LEADER]: Becoming Leader. State: Replica: 75aa2b7b902c4d14ad0b97ba92fb33bc, State: Running, Role: LEADER
I20260619 12:39:41.641834 21289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75aa2b7b902c4d14ad0b97ba92fb33bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } }
I20260619 12:39:41.642522 21279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:41.643678 21291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75aa2b7b902c4d14ad0b97ba92fb33bc. Latest consensus state: current_term: 1 leader_uuid: "75aa2b7b902c4d14ad0b97ba92fb33bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75aa2b7b902c4d14ad0b97ba92fb33bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } } }
I20260619 12:39:41.643814 21291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:41.644124 21290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75aa2b7b902c4d14ad0b97ba92fb33bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75aa2b7b902c4d14ad0b97ba92fb33bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } } }
I20260619 12:39:41.644212 21298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:41.644289 21290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:41.647389 21298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:41.653185 21298 catalog_manager.cc:1347] Generated new cluster ID: 7a2a4556dee345269614e11470682150
I20260619 12:39:41.653355 21298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:41.663718 21298 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:41.664099 21298 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:41.676689 21298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75aa2b7b902c4d14ad0b97ba92fb33bc: Generated new TSK 0
I20260619 12:39:41.677584 21298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:41.683008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 21211
2026-06-19T12:39:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:41 dist-test-slave-kdrb krb5kdc[21310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:41 dist-test-slave-kdrb krb5kdc[21310](info): set up 2 sockets
Jun 19 12:39:41 dist-test-slave-kdrb krb5kdc[21310](info): commencing operation
krb5kdc: starting...
W20260619 12:39:42.724143 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:42 dist-test-slave-kdrb krb5kdc[21310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:39:42.850975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34215
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:42.963057 21323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:42.963423 21323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:42.963533 21323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:42.967779 21323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:42.967900 21323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:42.967937 21323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:42.967976 21323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:42.968014 21323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:42.973809 21323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:42.974982 21323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:42.976092 21323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:42.984632 21328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:42.984654 21331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:42.984659 21329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:42.984695 21323 server_base.cc:1034] running on GCE node
I20260619 12:39:42.985437 21323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:42.986568 21323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:42.987728 21323 hybrid_clock.cc:648] HybridClock initialized: now 1781872782987695 us; error 49 us; skew 500 ppm
Jun 19 12:39:42 dist-test-slave-kdrb krb5kdc[21310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:42.990499 21323 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:39:42.991556 21323 webserver.cc:460] Webserver started at http://127.0.0.1:41659/ using document root <none> and password file <none>
I20260619 12:39:42.992173 21323 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:42.992272 21323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:42.992532 21323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:42.994681 21323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "988ef88fdbcc49adb5437b1c4b6a4e3e"
format_stamp: "Formatted at 2026-06-19 12:39:42 on dist-test-slave-kdrb"
server_key: "79a4ebd855b5f0e5a983b08c13681e4a"
server_key_iv: "7cb22ad2cd036cf9e343d61c397b0b93"
server_key_version: "encryptionkey@0"
I20260619 12:39:42.995296 21323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "988ef88fdbcc49adb5437b1c4b6a4e3e"
format_stamp: "Formatted at 2026-06-19 12:39:42 on dist-test-slave-kdrb"
server_key: "79a4ebd855b5f0e5a983b08c13681e4a"
server_key_iv: "7cb22ad2cd036cf9e343d61c397b0b93"
server_key_version: "encryptionkey@0"
I20260619 12:39:42.999287 21323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:39:43.001853 21338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:43.003161 21323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:39:43.003317 21323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "988ef88fdbcc49adb5437b1c4b6a4e3e"
format_stamp: "Formatted at 2026-06-19 12:39:42 on dist-test-slave-kdrb"
server_key: "79a4ebd855b5f0e5a983b08c13681e4a"
server_key_iv: "7cb22ad2cd036cf9e343d61c397b0b93"
server_key_version: "encryptionkey@0"
I20260619 12:39:43.003438 21323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:43.022324 21323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:43.025581 21323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:43.025808 21323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:43.033097 21323 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34215
I20260619 12:39:43.033124 21390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34215 every 8 connection(s)
I20260619 12:39:43.034083 21323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:43.037048 21391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:43.037988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 21323
I20260619 12:39:43.038143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:43.038475 392 external_mini_cluster.cc:1428] Setting key 538ec1f27f9fdacf83a99aa639423460
I20260619 12:39:43.043365 21391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e: Bootstrap starting.
I20260619 12:39:43.045706 21391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:43.046452 21391 log.cc:824] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:39:43 dist-test-slave-kdrb krb5kdc[21310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 12:39:43.048666 21391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e: No bootstrap required, opened a new log
I20260619 12:39:43.052531 21391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988ef88fdbcc49adb5437b1c4b6a4e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34215 } }
I20260619 12:39:43.052779 21391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:43.052850 21391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 988ef88fdbcc49adb5437b1c4b6a4e3e, State: Initialized, Role: FOLLOWER
I20260619 12:39:43.052968 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:43.039995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:50177 (local address 127.0.0.1:34215)
0619 12:39:43.040444 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:43.040459 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:43.040494 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:39:43.041127 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:43.041146 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:43.041525 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:43.041869 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:43.042654 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:43.043706 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:43.044557 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:43.045031 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:43.045204 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:43.048009 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:43.048014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:43.048028 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:43.048054 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:43.049838 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:43.050379 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:43.050385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:43.050390 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:43.050502 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:43.050840 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:43.050844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:43.050848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:43.051256 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:43.051520 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:43.051855 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 12:39:43.052086 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":138,"threads_started":1}
I20260619 12:39:43.053403 21391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988ef88fdbcc49adb5437b1c4b6a4e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34215 } }
I20260619 12:39:43.053567 21391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:43.053655 21391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:43.053783 21391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:43.055058 21391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988ef88fdbcc49adb5437b1c4b6a4e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34215 } }
I20260619 12:39:43.055436 21391 leader_election.cc:302] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 988ef88fdbcc49adb5437b1c4b6a4e3e; no voters:
I20260619 12:39:43.055835 21391 leader_election.cc:288] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:43.055935 21396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:43.056159 21396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [term 1 LEADER]: Becoming Leader. State: Replica: 988ef88fdbcc49adb5437b1c4b6a4e3e, State: Running, Role: LEADER
I20260619 12:39:43.056371 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:43.056521 21396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988ef88fdbcc49adb5437b1c4b6a4e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34215 } }
I20260619 12:39:43.057134 21391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:43.057829 21396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 988ef88fdbcc49adb5437b1c4b6a4e3e. Latest consensus state: current_term: 1 leader_uuid: "988ef88fdbcc49adb5437b1c4b6a4e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988ef88fdbcc49adb5437b1c4b6a4e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34215 } } }
I20260619 12:39:43.057937 21396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:43.058842 21397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "988ef88fdbcc49adb5437b1c4b6a4e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988ef88fdbcc49adb5437b1c4b6a4e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34215 } } }
I20260619 12:39:43.058952 21397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:43.059208 21409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:43.062570 21409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:43.068570 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:43.058691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:58689 (local address 127.0.0.1:34215)
0619 12:39:43.058813 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:43.058818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:43.059562 (+ 744us) server_negotiation.cc:407] Connection header received
0619 12:39:43.059758 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:43.059762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:43.059824 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:43.059905 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:43.060785 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:43.061586 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:43.062655 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:43.062986 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:43.063100 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:43.064812 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:43.064815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:43.064818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:43.064843 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:43.066350 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:43.066827 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:43.066831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:43.066833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:43.066916 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:43.067241 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:43.067245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:43.067247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:43.067429 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:43.067617 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:43.068323 (+ 706us) server_negotiation.cc:299] Negotiation successful
0619 12:39:43.068419 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 12:39:43.069481 21409 catalog_manager.cc:1347] Generated new cluster ID: e49a8079ce3c45b58c40bba13700e9da
I20260619 12:39:43.069614 21409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:43.106410 21409 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:43.107631 21409 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:43.113934 21409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 988ef88fdbcc49adb5437b1c4b6a4e3e: Generated new TSK 0
I20260619 12:39:43.114701 21409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:43.121572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 21323
W20260619 12:39:43.126523 21308 connection.cc:535] client connection to 127.0.0.1:34215 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34215): Connection reset by peer (error 104)
2026-06-19T12:39:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:43 dist-test-slave-kdrb krb5kdc[21422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:43 dist-test-slave-kdrb krb5kdc[21422](info): set up 2 sockets
Jun 19 12:39:43 dist-test-slave-kdrb krb5kdc[21422](info): commencing operation
krb5kdc: starting...
W20260619 12:39:44.159730 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:44 dist-test-slave-kdrb krb5kdc[21422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:39:44.287559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:44.399271 21435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:44.399682 21435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:44.399776 21435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:44.403865 21435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:44.403967 21435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:44.404006 21435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:44.404044 21435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:44.404078 21435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:44.409644 21435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:44.410817 21435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:44.411921 21435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:44.420274 21440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:44.420429 21435 server_base.cc:1034] running on GCE node
W20260619 12:39:44.420308 21441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:44.420606 21443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:44.421164 21435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:44.422379 21435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:44.423547 21435 hybrid_clock.cc:648] HybridClock initialized: now 1781872784423528 us; error 40 us; skew 500 ppm
Jun 19 12:39:44 dist-test-slave-kdrb krb5kdc[21422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:44.426383 21435 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:39:44.427314 21435 webserver.cc:460] Webserver started at http://127.0.0.1:36171/ using document root <none> and password file <none>
I20260619 12:39:44.427953 21435 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:44.428057 21435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:44.428296 21435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:44.430402 21435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "14176499204748f5a746adb577f487b8"
format_stamp: "Formatted at 2026-06-19 12:39:44 on dist-test-slave-kdrb"
server_key: "bfc018eba15f3f7623a40e82a2103cf4"
server_key_iv: "e5bf71eecdb036b814fcd7b544aeac7c"
server_key_version: "encryptionkey@0"
I20260619 12:39:44.430961 21435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "14176499204748f5a746adb577f487b8"
format_stamp: "Formatted at 2026-06-19 12:39:44 on dist-test-slave-kdrb"
server_key: "bfc018eba15f3f7623a40e82a2103cf4"
server_key_iv: "e5bf71eecdb036b814fcd7b544aeac7c"
server_key_version: "encryptionkey@0"
I20260619 12:39:44.434875 21435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:39:44.437415 21450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:44.438735 21435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:39:44.438908 21435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "14176499204748f5a746adb577f487b8"
format_stamp: "Formatted at 2026-06-19 12:39:44 on dist-test-slave-kdrb"
server_key: "bfc018eba15f3f7623a40e82a2103cf4"
server_key_iv: "e5bf71eecdb036b814fcd7b544aeac7c"
server_key_version: "encryptionkey@0"
I20260619 12:39:44.439040 21435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:44.456821 21435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:44.460269 21435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:44.460494 21435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:44.467372 21435 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46053
I20260619 12:39:44.467453 21502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46053 every 8 connection(s)
I20260619 12:39:44.468391 21435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:44.471122 21503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:44.474025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 21435
I20260619 12:39:44.474184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:44.474462 392 external_mini_cluster.cc:1428] Setting key 95ea32c18b75155c098e24a8883a16de
I20260619 12:39:44.476936 21503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8: Bootstrap starting.
I20260619 12:39:44.479480 21503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:44.480388 21503 log.cc:824] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:44.482635 21503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8: No bootstrap required, opened a new log
I20260619 12:39:44.483839 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:44.475790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:46135 (local address 127.0.0.1:46053)
0619 12:39:44.476205 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:44.476220 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:44.476255 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:39:44.476846 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:44.476862 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:44.477244 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:44.477531 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:44.477847 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:44.478804 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:44.480110 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:44.480677 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:44.480884 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:44.481864 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:44.481870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:44.481883 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:44.482302 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:44.482365 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:44.482392 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:44.482548 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:44.482764 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 12:39:44.483058 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":138,"threads_started":1}
I20260619 12:39:44.486768 21503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14176499204748f5a746adb577f487b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46053 } }
I20260619 12:39:44.487015 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:44.487056 21503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:44.487139 21503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14176499204748f5a746adb577f487b8, State: Initialized, Role: FOLLOWER
I20260619 12:39:44.487634 21503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14176499204748f5a746adb577f487b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46053 } }
I20260619 12:39:44.487794 21503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:44.487876 21503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:44.487974 21503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:44.488994 21503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14176499204748f5a746adb577f487b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46053 } }
I20260619 12:39:44.489358 21503 leader_election.cc:302] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14176499204748f5a746adb577f487b8; no voters:
I20260619 12:39:44.489677 21503 leader_election.cc:288] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:44.489811 21513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:44.490063 21513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [term 1 LEADER]: Becoming Leader. State: Replica: 14176499204748f5a746adb577f487b8, State: Running, Role: LEADER
I20260619 12:39:44.490449 21513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14176499204748f5a746adb577f487b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46053 } }
I20260619 12:39:44.491009 21503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:44.492275 21514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14176499204748f5a746adb577f487b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14176499204748f5a746adb577f487b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46053 } } }
I20260619 12:39:44.492275 21515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14176499204748f5a746adb577f487b8. Latest consensus state: current_term: 1 leader_uuid: "14176499204748f5a746adb577f487b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14176499204748f5a746adb577f487b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46053 } } }
I20260619 12:39:44.492448 21515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:44.492448 21514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:44.492942 21523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:44.494670 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:44.489440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:46093 (local address 127.0.0.1:46053)
0619 12:39:44.489553 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:44.489557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:44.489779 (+ 222us) server_negotiation.cc:407] Connection header received
0619 12:39:44.490004 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:44.490008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:44.490056 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:44.490136 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:44.490526 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:44.491250 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:44.492480 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:44.492874 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:44.492971 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:44.494137 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:44.494143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:44.494147 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:44.494165 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:44.494196 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:44.494209 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:44.494289 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:44.494496 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 12:39:44.494531 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:39:44.496189 21523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:44.501547 21523 catalog_manager.cc:1347] Generated new cluster ID: 90bed72317284c5db083f24dad5a4f39
I20260619 12:39:44.501632 21523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:44.512288 21523 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:44.513324 21523 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:44.519460 21523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14176499204748f5a746adb577f487b8: Generated new TSK 0
I20260619 12:39:44.520195 21523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:44.527590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 21435
W20260619 12:39:44.532943 21420 connection.cc:535] client connection to 127.0.0.1:46053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46053): Connection reset by peer (error 104)
2026-06-19T12:39:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:44 dist-test-slave-kdrb krb5kdc[21534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:44 dist-test-slave-kdrb krb5kdc[21534](info): set up 2 sockets
Jun 19 12:39:44 dist-test-slave-kdrb krb5kdc[21534](info): commencing operation
krb5kdc: starting...
W20260619 12:39:45.567734 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:45 dist-test-slave-kdrb krb5kdc[21534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:39:45.696812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:45.808873 21547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:45.809190 21547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:45.809233 21547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:45.813475 21547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:45.813552 21547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:45.813572 21547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:45.813589 21547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:45.813654 21547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:45.819183 21547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:45.820393 21547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:45.821485 21547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:45.830364 21552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:45.830350 21555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:45.830314 21553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:45.830411 21547 server_base.cc:1034] running on GCE node
I20260619 12:39:45.831286 21547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:45.832568 21547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:45.833799 21547 hybrid_clock.cc:648] HybridClock initialized: now 1781872785833781 us; error 33 us; skew 500 ppm
Jun 19 12:39:45 dist-test-slave-kdrb krb5kdc[21534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:45.841452 21547 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:39:45.842584 21547 webserver.cc:460] Webserver started at http://127.0.0.1:45385/ using document root <none> and password file <none>
I20260619 12:39:45.843222 21547 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:45.843331 21547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:45.843647 21547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:45.845878 21547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "a333c0819cfb485db16227d24118cb23"
format_stamp: "Formatted at 2026-06-19 12:39:45 on dist-test-slave-kdrb"
server_key: "4bac83b53352fe1d02bf4aed85311340"
server_key_iv: "e2a012b7330720a7482bb80add5f376a"
server_key_version: "encryptionkey@0"
I20260619 12:39:45.846531 21547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "a333c0819cfb485db16227d24118cb23"
format_stamp: "Formatted at 2026-06-19 12:39:45 on dist-test-slave-kdrb"
server_key: "4bac83b53352fe1d02bf4aed85311340"
server_key_iv: "e2a012b7330720a7482bb80add5f376a"
server_key_version: "encryptionkey@0"
I20260619 12:39:45.850593 21547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:39:45.853384 21562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:45.854753 21547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:39:45.854918 21547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "a333c0819cfb485db16227d24118cb23"
format_stamp: "Formatted at 2026-06-19 12:39:45 on dist-test-slave-kdrb"
server_key: "4bac83b53352fe1d02bf4aed85311340"
server_key_iv: "e2a012b7330720a7482bb80add5f376a"
server_key_version: "encryptionkey@0"
I20260619 12:39:45.855049 21547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:45.860049 21547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:45.864265 21547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:45.864495 21547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:45.871814 21547 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43997
I20260619 12:39:45.871891 21614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43997 every 8 connection(s)
I20260619 12:39:45.872937 21547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:45.873389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 21547
I20260619 12:39:45.873558 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:45.873860 392 external_mini_cluster.cc:1428] Setting key 6186a99f1978d437289560c7af1b396a
I20260619 12:39:45.876856 21615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:45.880105 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:45.875288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:39307 (local address 127.0.0.1:43997)
0619 12:39:45.875810 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:45.875830 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:45.875874 (+ 44us) server_negotiation.cc:407] Connection header received
0619 12:39:45.876778 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:45.876804 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:45.877205 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:45.877607 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:45.878527 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:45.878532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:45.878549 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:45.878561 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:45.878633 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:45.878662 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:45.878816 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:45.878927 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 12:39:45.879089 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":176,"threads_started":1}
I20260619 12:39:45.883256 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:45.884238 21615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23: Bootstrap starting.
I20260619 12:39:45.886840 21615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:45.887374 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:45.885536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:47859 (local address 127.0.0.1:43997)
0619 12:39:45.885680 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:45.885684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:45.885898 (+ 214us) server_negotiation.cc:407] Connection header received
0619 12:39:45.886058 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:45.886061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:45.886105 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:45.886193 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:45.886981 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:45.886985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:45.886987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:45.886998 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:45.887031 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:45.887042 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:45.887133 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:45.887206 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 12:39:45.887251 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 12:39:45.887858 21615 log.cc:824] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:45.890339 21615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23: No bootstrap required, opened a new log
I20260619 12:39:45.894299 21615 raft_consensus.cc:348] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a333c0819cfb485db16227d24118cb23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43997 } }
I20260619 12:39:45.894641 21615 raft_consensus.cc:374] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:45.894737 21615 raft_consensus.cc:729] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a333c0819cfb485db16227d24118cb23, State: Initialized, Role: FOLLOWER
I20260619 12:39:45.895244 21615 consensus_queue.cc:260] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a333c0819cfb485db16227d24118cb23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43997 } }
I20260619 12:39:45.895388 21615 raft_consensus.cc:388] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:45.895432 21615 raft_consensus.cc:482] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:45.895573 21615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:45.896625 21615 raft_consensus.cc:504] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a333c0819cfb485db16227d24118cb23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43997 } }
I20260619 12:39:45.896967 21615 leader_election.cc:302] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a333c0819cfb485db16227d24118cb23; no voters:
I20260619 12:39:45.897264 21615 leader_election.cc:288] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:45.897415 21625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:45.897691 21625 raft_consensus.cc:686] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [term 1 LEADER]: Becoming Leader. State: Replica: a333c0819cfb485db16227d24118cb23, State: Running, Role: LEADER
I20260619 12:39:45.898051 21625 consensus_queue.cc:237] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a333c0819cfb485db16227d24118cb23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43997 } }
I20260619 12:39:45.898609 21615 sys_catalog.cc:564] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:45.899869 21627 sys_catalog.cc:455] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a333c0819cfb485db16227d24118cb23. Latest consensus state: current_term: 1 leader_uuid: "a333c0819cfb485db16227d24118cb23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a333c0819cfb485db16227d24118cb23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43997 } } }
I20260619 12:39:45.900007 21627 sys_catalog.cc:458] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:45.900453 21626 sys_catalog.cc:455] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a333c0819cfb485db16227d24118cb23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a333c0819cfb485db16227d24118cb23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43997 } } }
I20260619 12:39:45.900599 21626 sys_catalog.cc:458] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23 [sys.catalog]: This master's current role is: LEADER
W20260619 12:39:45.903862 21640 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:39:45.903965 21640 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:39:45.904053 21634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:45.904776 21634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:45.910637 21634 catalog_manager.cc:1347] Generated new cluster ID: 6da6ef3b88c346488e36fcbe2df27189
I20260619 12:39:45.910727 21634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:45.920835 21634 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:45.921227 21634 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:45.931900 21634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a333c0819cfb485db16227d24118cb23: Generated new TSK 0
I20260619 12:39:45.932699 21634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:45.943231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 21547
2026-06-19T12:39:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:46 dist-test-slave-kdrb krb5kdc[21646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:46 dist-test-slave-kdrb krb5kdc[21646](info): set up 2 sockets
Jun 19 12:39:46 dist-test-slave-kdrb krb5kdc[21646](info): commencing operation
krb5kdc: starting...
W20260619 12:39:46.987752 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:47 dist-test-slave-kdrb krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:39:47.114480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33095
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:47.228443 21659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:47.228806 21659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:47.228897 21659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:47.232980 21659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:47.233084 21659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:47.233124 21659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:47.233160 21659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:47.233199 21659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:47.238627 21659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33095
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:47.239836 21659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:47.240906 21659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:39:47.249930 21659 server_base.cc:1034] running on GCE node
W20260619 12:39:47.249873 21664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:47.249867 21665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:47.249873 21667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:47.250835 21659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:47.252161 21659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:47.253345 21659 hybrid_clock.cc:648] HybridClock initialized: now 1781872787253306 us; error 60 us; skew 500 ppm
Jun 19 12:39:47 dist-test-slave-kdrb krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:47.256513 21659 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:39:47.257594 21659 webserver.cc:460] Webserver started at http://127.0.0.1:43057/ using document root <none> and password file <none>
I20260619 12:39:47.258209 21659 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:47.258306 21659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:47.258553 21659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:47.260878 21659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "979abe2bb87f4974923bae0934190f32"
format_stamp: "Formatted at 2026-06-19 12:39:47 on dist-test-slave-kdrb"
server_key: "7664ed1001779f2494e8cdd19d93d56d"
server_key_iv: "88299358eff4fef7360289b7e8854a8a"
server_key_version: "encryptionkey@0"
I20260619 12:39:47.261512 21659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "979abe2bb87f4974923bae0934190f32"
format_stamp: "Formatted at 2026-06-19 12:39:47 on dist-test-slave-kdrb"
server_key: "7664ed1001779f2494e8cdd19d93d56d"
server_key_iv: "88299358eff4fef7360289b7e8854a8a"
server_key_version: "encryptionkey@0"
I20260619 12:39:47.265538 21659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:39:47.268194 21674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:47.269608 21659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:47.269779 21659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "979abe2bb87f4974923bae0934190f32"
format_stamp: "Formatted at 2026-06-19 12:39:47 on dist-test-slave-kdrb"
server_key: "7664ed1001779f2494e8cdd19d93d56d"
server_key_iv: "88299358eff4fef7360289b7e8854a8a"
server_key_version: "encryptionkey@0"
I20260619 12:39:47.269910 21659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:47.294559 21659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:47.298204 21659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:47.298434 21659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:47.306411 21659 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33095
I20260619 12:39:47.306420 21726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33095 every 8 connection(s)
I20260619 12:39:47.307569 21659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:47.310478 21727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:47.311553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 21659
I20260619 12:39:47.311702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:47.312045 392 external_mini_cluster.cc:1428] Setting key 5c4ec73a2b5db50ebec2e7fbb7b9ff47
I20260619 12:39:47.316991 21727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32: Bootstrap starting.
I20260619 12:39:47.319600 21727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:47.320359 21727 log.cc:824] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:39:47 dist-test-slave-kdrb krb5kdc[21646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 12:39:47.322412 21727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32: No bootstrap required, opened a new log
I20260619 12:39:47.326015 21727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979abe2bb87f4974923bae0934190f32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33095 } }
I20260619 12:39:47.326347 21727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:47.326411 21727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979abe2bb87f4974923bae0934190f32, State: Initialized, Role: FOLLOWER
I20260619 12:39:47.326913 21730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:47.313505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:38725 (local address 127.0.0.1:33095)
0619 12:39:47.313979 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:47.313995 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:47.314031 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:39:47.314677 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:47.314694 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:47.315050 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:47.315405 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:47.316240 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:47.317298 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:47.318441 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:47.318946 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:47.319126 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:47.321892 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:47.321905 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:47.321920 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:47.321947 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:47.323817 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:47.324359 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:47.324364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:47.324369 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:47.324480 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:47.324829 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:47.324832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:47.324836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:47.325240 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:47.325480 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:47.325768 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 12:39:47.325996 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":146,"threads_started":1}
I20260619 12:39:47.326884 21727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979abe2bb87f4974923bae0934190f32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33095 } }
I20260619 12:39:47.327050 21727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:47.327153 21727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:47.327296 21727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:47.328512 21727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979abe2bb87f4974923bae0934190f32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33095 } }
I20260619 12:39:47.328917 21727 leader_election.cc:302] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 979abe2bb87f4974923bae0934190f32; no voters:
I20260619 12:39:47.329288 21727 leader_election.cc:288] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:47.329424 21732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:47.329604 21732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [term 1 LEADER]: Becoming Leader. State: Replica: 979abe2bb87f4974923bae0934190f32, State: Running, Role: LEADER
I20260619 12:39:47.330157 21732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979abe2bb87f4974923bae0934190f32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33095 } }
I20260619 12:39:47.330384 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:47.330655 21727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:47.332154 21734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "979abe2bb87f4974923bae0934190f32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979abe2bb87f4974923bae0934190f32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33095 } } }
I20260619 12:39:47.332211 21733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 979abe2bb87f4974923bae0934190f32. Latest consensus state: current_term: 1 leader_uuid: "979abe2bb87f4974923bae0934190f32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979abe2bb87f4974923bae0934190f32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33095 } } }
I20260619 12:39:47.332295 21734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:47.332317 21733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:47.332896 21746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:47.336278 21746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:47.343170 21746 catalog_manager.cc:1347] Generated new cluster ID: 623fe2631c2c4197b3c119c4a3e25b5f
I20260619 12:39:47.343322 21746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:47.344002 21730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:47.333612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:49705 (local address 127.0.0.1:33095)
0619 12:39:47.333750 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:47.333755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:47.334149 (+ 394us) server_negotiation.cc:407] Connection header received
0619 12:39:47.334493 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:47.334496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:47.334547 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:47.334639 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:47.335551 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:47.336270 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:47.337602 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:47.337951 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:47.338061 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:47.339838 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:47.339842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:47.339845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:47.339875 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:47.341840 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:47.342485 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:47.342489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:47.342491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:47.342583 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:47.342946 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:47.342949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:47.342950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:47.343183 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:47.343416 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:47.343744 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 12:39:47.343844 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 12:39:47.358449 21746 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:47.359544 21746 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:47.366061 21746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 979abe2bb87f4974923bae0934190f32: Generated new TSK 0
I20260619 12:39:47.366847 21746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:47.375299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 21659
W20260619 12:39:47.380858 21644 connection.cc:535] client connection to 127.0.0.1:33095 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33095): Connection reset by peer (error 104)
2026-06-19T12:39:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:47 dist-test-slave-kdrb krb5kdc[21758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:47 dist-test-slave-kdrb krb5kdc[21758](info): set up 2 sockets
Jun 19 12:39:47 dist-test-slave-kdrb krb5kdc[21758](info): commencing operation
krb5kdc: starting...
W20260619 12:39:48.431839 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:48 dist-test-slave-kdrb krb5kdc[21758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:39:48.563055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:48.676697 21771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:48.677091 21771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:48.677181 21771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:48.681371 21771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:48.681499 21771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:48.681538 21771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:48.681573 21771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:48.681604 21771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:48.687220 21771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:48.688474 21771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:48.689568 21771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:48.698845 21779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:48.698801 21777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:48.698817 21776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:48.698889 21771 server_base.cc:1034] running on GCE node
I20260619 12:39:48.699731 21771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:48.701028 21771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:48.702211 21771 hybrid_clock.cc:648] HybridClock initialized: now 1781872788702188 us; error 37 us; skew 500 ppm
Jun 19 12:39:48 dist-test-slave-kdrb krb5kdc[21758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:48.705245 21771 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:39:48.706209 21771 webserver.cc:460] Webserver started at http://127.0.0.1:39119/ using document root <none> and password file <none>
I20260619 12:39:48.706813 21771 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:48.706914 21771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:48.707162 21771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:48.709471 21771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "710aa8d7a9604a05a24968e9ce496907"
format_stamp: "Formatted at 2026-06-19 12:39:48 on dist-test-slave-kdrb"
server_key: "3f9df7d6b452dbd3a1713a6b6501e4fb"
server_key_iv: "5e3bb84bac46f9e98c3d32b06737968a"
server_key_version: "encryptionkey@0"
I20260619 12:39:48.710053 21771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "710aa8d7a9604a05a24968e9ce496907"
format_stamp: "Formatted at 2026-06-19 12:39:48 on dist-test-slave-kdrb"
server_key: "3f9df7d6b452dbd3a1713a6b6501e4fb"
server_key_iv: "5e3bb84bac46f9e98c3d32b06737968a"
server_key_version: "encryptionkey@0"
I20260619 12:39:48.714037 21771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:39:48.716742 21786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:48.718012 21771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:39:48.718176 21771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "710aa8d7a9604a05a24968e9ce496907"
format_stamp: "Formatted at 2026-06-19 12:39:48 on dist-test-slave-kdrb"
server_key: "3f9df7d6b452dbd3a1713a6b6501e4fb"
server_key_iv: "5e3bb84bac46f9e98c3d32b06737968a"
server_key_version: "encryptionkey@0"
I20260619 12:39:48.718308 21771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:48.730974 21771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:48.734649 21771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:48.734877 21771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:48.742151 21771 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42113
I20260619 12:39:48.742177 21838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42113 every 8 connection(s)
I20260619 12:39:48.743139 21771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:48.745960 21839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:48.749740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 21771
I20260619 12:39:48.749927 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:48.750209 392 external_mini_cluster.cc:1428] Setting key 15b7ddfc9e78f1f98b5b10414f2bced1
I20260619 12:39:48.752076 21839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907: Bootstrap starting.
I20260619 12:39:48.754948 21839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:48.755945 21839 log.cc:824] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:48.757972 21839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907: No bootstrap required, opened a new log
I20260619 12:39:48.760790 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:48.751748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:49221 (local address 127.0.0.1:42113)
0619 12:39:48.752535 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:48.752555 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:48.752606 (+ 51us) server_negotiation.cc:407] Connection header received
0619 12:39:48.753501 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:48.753525 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:48.753974 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:48.754365 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:48.754792 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:48.756382 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:48.757390 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:48.757893 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:48.758077 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:48.759001 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:48.759006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:48.759020 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:48.759428 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:48.759540 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:48.759573 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:48.759718 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:48.759911 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 12:39:48.760059 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":240,"threads_started":1}
I20260619 12:39:48.762120 21839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710aa8d7a9604a05a24968e9ce496907" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } }
I20260619 12:39:48.762390 21839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:48.762442 21839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710aa8d7a9604a05a24968e9ce496907, State: Initialized, Role: FOLLOWER
I20260619 12:39:48.762903 21839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710aa8d7a9604a05a24968e9ce496907" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } }
I20260619 12:39:48.763038 21839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:48.763125 21839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:48.763204 21839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:48.763968 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:48.764333 21839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710aa8d7a9604a05a24968e9ce496907" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } }
I20260619 12:39:48.764675 21839 leader_election.cc:302] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 710aa8d7a9604a05a24968e9ce496907; no voters:
I20260619 12:39:48.765033 21839 leader_election.cc:288] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:48.765108 21844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:48.765290 21844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [term 1 LEADER]: Becoming Leader. State: Replica: 710aa8d7a9604a05a24968e9ce496907, State: Running, Role: LEADER
I20260619 12:39:48.765600 21844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710aa8d7a9604a05a24968e9ce496907" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } }
I20260619 12:39:48.766335 21839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:48.767144 21847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "710aa8d7a9604a05a24968e9ce496907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710aa8d7a9604a05a24968e9ce496907" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } } }
I20260619 12:39:48.767308 21847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:48.767654 21848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 710aa8d7a9604a05a24968e9ce496907. Latest consensus state: current_term: 1 leader_uuid: "710aa8d7a9604a05a24968e9ce496907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710aa8d7a9604a05a24968e9ce496907" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } } }
I20260619 12:39:48.767743 21848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:48.768194 21858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:48.771078 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:48.766106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:39737 (local address 127.0.0.1:42113)
0619 12:39:48.766228 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:48.766232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:48.766614 (+ 382us) server_negotiation.cc:407] Connection header received
0619 12:39:48.766657 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:48.766661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:48.766722 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:48.766792 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:48.767167 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:48.767972 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:48.769099 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:48.769420 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:48.769541 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:48.770535 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:48.770539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:48.770542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:48.770555 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:48.770585 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:48.770599 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:48.770674 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:48.770886 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 12:39:48.770922 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 12:39:48.771508 21858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:48.777205 21858 catalog_manager.cc:1347] Generated new cluster ID: 4b9536344343441380f905e87c68d477
I20260619 12:39:48.777289 21858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:48.795557 21858 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:48.796758 21858 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:48.803756 21858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 710aa8d7a9604a05a24968e9ce496907: Generated new TSK 0
I20260619 12:39:48.804507 21858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:48.816838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 21771
W20260619 12:39:48.822285 21756 connection.cc:535] client connection to 127.0.0.1:42113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42113): Connection reset by peer (error 104)
2026-06-19T12:39:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:48 dist-test-slave-kdrb krb5kdc[21870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:48 dist-test-slave-kdrb krb5kdc[21870](info): set up 2 sockets
Jun 19 12:39:48 dist-test-slave-kdrb krb5kdc[21870](info): commencing operation
krb5kdc: starting...
W20260619 12:39:49.859764 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:49 dist-test-slave-kdrb krb5kdc[21870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:39:49.990566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:50.103340 21883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:50.103785 21883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:50.103888 21883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:50.108036 21883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:50.108158 21883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:50.108197 21883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:50.108234 21883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:50.108268 21883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:50.114002 21883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44831
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:50.115240 21883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:50.116549 21883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:50.126114 21888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:50.126132 21889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:50.126132 21891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:50.126267 21883 server_base.cc:1034] running on GCE node
I20260619 12:39:50.127000 21883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:50.128228 21883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:50.129401 21883 hybrid_clock.cc:648] HybridClock initialized: now 1781872790129372 us; error 48 us; skew 500 ppm
Jun 19 12:39:50 dist-test-slave-kdrb krb5kdc[21870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:50.132365 21883 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:39:50.133360 21883 webserver.cc:460] Webserver started at http://127.0.0.1:40367/ using document root <none> and password file <none>
I20260619 12:39:50.133956 21883 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:50.134056 21883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:50.134331 21883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:50.136633 21883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "4a9aa6ad923e45a48b2498d4a5e98adb"
format_stamp: "Formatted at 2026-06-19 12:39:50 on dist-test-slave-kdrb"
server_key: "2aada6b378aac9167bddc5ff632270ac"
server_key_iv: "909b574f155f5b8efafe2fe093905c23"
server_key_version: "encryptionkey@0"
I20260619 12:39:50.137228 21883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a9aa6ad923e45a48b2498d4a5e98adb"
format_stamp: "Formatted at 2026-06-19 12:39:50 on dist-test-slave-kdrb"
server_key: "2aada6b378aac9167bddc5ff632270ac"
server_key_iv: "909b574f155f5b8efafe2fe093905c23"
server_key_version: "encryptionkey@0"
I20260619 12:39:50.141234 21883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:39:50.143939 21898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:50.145334 21883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:50.145509 21883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "4a9aa6ad923e45a48b2498d4a5e98adb"
format_stamp: "Formatted at 2026-06-19 12:39:50 on dist-test-slave-kdrb"
server_key: "2aada6b378aac9167bddc5ff632270ac"
server_key_iv: "909b574f155f5b8efafe2fe093905c23"
server_key_version: "encryptionkey@0"
I20260619 12:39:50.145650 21883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:50.150574 21883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:50.154014 21883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:50.154232 21883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:50.162143 21883 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44831
I20260619 12:39:50.162161 21950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44831 every 8 connection(s)
I20260619 12:39:50.163189 21883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:50.166250 21951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:50.167071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 21883
I20260619 12:39:50.167232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:50.167615 392 external_mini_cluster.cc:1428] Setting key 00878c995280e33c51f7efd549085a86
I20260619 12:39:50.173305 21951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb: Bootstrap starting.
I20260619 12:39:50.173367 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:50.169231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:33099 (local address 127.0.0.1:44831)
0619 12:39:50.169749 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:50.169765 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:50.169808 (+ 43us) server_negotiation.cc:407] Connection header received
0619 12:39:50.170469 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:50.170488 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:50.170798 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:50.171194 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:50.172041 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:50.172046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:50.172071 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:50.172081 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:50.172136 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:50.172155 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:50.172329 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:50.172432 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 12:39:50.172602 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":159,"threads_started":1}
I20260619 12:39:50.176222 21951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:50.176738 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:50.177168 21951 log.cc:824] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:50.179674 21951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb: No bootstrap required, opened a new log
I20260619 12:39:50.180860 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:50.179031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:39069 (local address 127.0.0.1:44831)
0619 12:39:50.179159 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:50.179168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:50.179405 (+ 237us) server_negotiation.cc:407] Connection header received
0619 12:39:50.179568 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:50.179571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:50.179605 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:50.179676 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:50.180419 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:50.180422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:50.180424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:50.180435 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:50.180460 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:50.180471 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:50.180541 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:50.180645 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 12:39:50.180679 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 12:39:50.184047 21951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9aa6ad923e45a48b2498d4a5e98adb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260619 12:39:50.184319 21951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:50.184420 21951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a9aa6ad923e45a48b2498d4a5e98adb, State: Initialized, Role: FOLLOWER
I20260619 12:39:50.184957 21951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9aa6ad923e45a48b2498d4a5e98adb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260619 12:39:50.185124 21951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:50.185200 21951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:50.185284 21951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:50.186332 21951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9aa6ad923e45a48b2498d4a5e98adb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260619 12:39:50.186769 21951 leader_election.cc:302] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a9aa6ad923e45a48b2498d4a5e98adb; no voters:
I20260619 12:39:50.187224 21951 leader_election.cc:288] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:50.187299 21961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:50.187515 21961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [term 1 LEADER]: Becoming Leader. State: Replica: 4a9aa6ad923e45a48b2498d4a5e98adb, State: Running, Role: LEADER
I20260619 12:39:50.187847 21961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9aa6ad923e45a48b2498d4a5e98adb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260619 12:39:50.188701 21951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:50.189813 21962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a9aa6ad923e45a48b2498d4a5e98adb. Latest consensus state: current_term: 1 leader_uuid: "4a9aa6ad923e45a48b2498d4a5e98adb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9aa6ad923e45a48b2498d4a5e98adb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } } }
I20260619 12:39:50.189855 21963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a9aa6ad923e45a48b2498d4a5e98adb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9aa6ad923e45a48b2498d4a5e98adb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } } }
I20260619 12:39:50.189996 21963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:50.189994 21962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:50.191417 21970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:50.195796 21970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:50.201869 21970 catalog_manager.cc:1347] Generated new cluster ID: e2cd9375466f4e27897410e701cd72e6
I20260619 12:39:50.201956 21970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:50.216257 21970 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:50.216637 21970 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:50.223251 21970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a9aa6ad923e45a48b2498d4a5e98adb: Generated new TSK 0
I20260619 12:39:50.224059 21970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:50.232299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 21883
2026-06-19T12:39:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:50 dist-test-slave-kdrb krb5kdc[21982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:50 dist-test-slave-kdrb krb5kdc[21982](info): set up 2 sockets
Jun 19 12:39:50 dist-test-slave-kdrb krb5kdc[21982](info): commencing operation
krb5kdc: starting...
W20260619 12:39:51.275770 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:51 dist-test-slave-kdrb krb5kdc[21982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:39:51.406311 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39731
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:51.520241 21995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:51.520603 21995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:51.520694 21995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:51.525036 21995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:51.525157 21995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:51.525202 21995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:51.525269 21995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:51.525312 21995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:51.531031 21995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39731
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:51.532388 21995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:51.533502 21995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:39:51.543020 21995 server_base.cc:1034] running on GCE node
W20260619 12:39:51.542954 22000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:51.542966 22001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:51.542959 22003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:51.543964 21995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:51.545128 21995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:51.546347 21995 hybrid_clock.cc:648] HybridClock initialized: now 1781872791546313 us; error 50 us; skew 500 ppm
Jun 19 12:39:51 dist-test-slave-kdrb krb5kdc[21982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:51.554692 21995 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:39:51.555887 21995 webserver.cc:460] Webserver started at http://127.0.98.62:42623/ using document root <none> and password file <none>
I20260619 12:39:51.556568 21995 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:51.556679 21995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:51.556936 21995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:51.559136 21995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "b7031c14c13d449e8cc69f505281e2be"
format_stamp: "Formatted at 2026-06-19 12:39:51 on dist-test-slave-kdrb"
server_key: "80d4c49559508abcdc1991c97bf629e9"
server_key_iv: "24426560c6051b5e8ea3d903b2b1cc4f"
server_key_version: "encryptionkey@0"
I20260619 12:39:51.559777 21995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "b7031c14c13d449e8cc69f505281e2be"
format_stamp: "Formatted at 2026-06-19 12:39:51 on dist-test-slave-kdrb"
server_key: "80d4c49559508abcdc1991c97bf629e9"
server_key_iv: "24426560c6051b5e8ea3d903b2b1cc4f"
server_key_version: "encryptionkey@0"
I20260619 12:39:51.563925 21995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 12:39:51.566632 22010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:51.568058 21995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:39:51.568222 21995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "b7031c14c13d449e8cc69f505281e2be"
format_stamp: "Formatted at 2026-06-19 12:39:51 on dist-test-slave-kdrb"
server_key: "80d4c49559508abcdc1991c97bf629e9"
server_key_iv: "24426560c6051b5e8ea3d903b2b1cc4f"
server_key_version: "encryptionkey@0"
I20260619 12:39:51.568576 21995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:51.586584 21995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:51.590422 21995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:51.590664 21995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:51.598279 21995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39731
I20260619 12:39:51.598304 22062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39731 every 8 connection(s)
I20260619 12:39:51.599349 21995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:51.602397 22063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:51.603178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 21995
I20260619 12:39:51.603327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:51.603675 392 external_mini_cluster.cc:1428] Setting key aafeeebf737aa096f633bbe351dc03c3
I20260619 12:39:51.608809 22063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be: Bootstrap starting.
I20260619 12:39:51.611122 22063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:51.611925 22063 log.cc:824] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:39:51 dist-test-slave-kdrb krb5kdc[21982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:39:51.614322 22063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be: No bootstrap required, opened a new log
I20260619 12:39:51.618886 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:51.605150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46465 (local address 127.0.98.62:39731)
0619 12:39:51.605635 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:51.605651 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:51.605688 (+ 37us) server_negotiation.cc:407] Connection header received
0619 12:39:51.606363 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:51.606380 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:51.606737 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:51.607101 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:51.607985 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:51.609008 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:51.609937 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:51.610438 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:51.610625 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:51.613664 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:51.613669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:51.613684 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:51.613711 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:51.615625 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:51.616265 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:51.616270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:51.616285 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:51.616408 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:51.616826 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:51.616828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:51.616829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:51.617237 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:51.617414 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:51.617770 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 12:39:51.618061 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":151,"threads_started":1}
I20260619 12:39:51.618857 22063 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7031c14c13d449e8cc69f505281e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39731 } }
I20260619 12:39:51.619189 22063 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:51.619261 22063 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7031c14c13d449e8cc69f505281e2be, State: Initialized, Role: FOLLOWER
I20260619 12:39:51.619908 22063 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7031c14c13d449e8cc69f505281e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39731 } }
I20260619 12:39:51.620141 22068 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:39:51.620275 22068 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7031c14c13d449e8cc69f505281e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39731 } }
I20260619 12:39:51.620095 22063 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:51.620708 22063 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:51.620704 22068 leader_election.cc:302] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7031c14c13d449e8cc69f505281e2be; no voters:
I20260619 12:39:51.620827 22063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:51.621104 22068 leader_election.cc:288] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260619 12:39:51.622155 22063 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7031c14c13d449e8cc69f505281e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39731 } }
I20260619 12:39:51.622318 22063 leader_election.cc:302] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7031c14c13d449e8cc69f505281e2be; no voters:
I20260619 12:39:51.622397 22068 raft_consensus.cc:2741] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 12:39:51.622406 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:51.622468 22063 leader_election.cc:288] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:51.622681 22069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:51.622947 22069 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [term 1 LEADER]: Becoming Leader. State: Replica: b7031c14c13d449e8cc69f505281e2be, State: Running, Role: LEADER
I20260619 12:39:51.623354 22069 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7031c14c13d449e8cc69f505281e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39731 } }
I20260619 12:39:51.623982 22063 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:51.624905 22068 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7031c14c13d449e8cc69f505281e2be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7031c14c13d449e8cc69f505281e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39731 } } }
I20260619 12:39:51.625030 22068 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:51.625229 22068 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7031c14c13d449e8cc69f505281e2be. Latest consensus state: current_term: 1 leader_uuid: "b7031c14c13d449e8cc69f505281e2be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7031c14c13d449e8cc69f505281e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39731 } } }
I20260619 12:39:51.625301 22068 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be [sys.catalog]: This master's current role is: LEADER
W20260619 12:39:51.629189 22089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:39:51.629299 22089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:39:51.629405 22082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:51.630201 22082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:51.636231 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:51.624832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49675 (local address 127.0.98.62:39731)
0619 12:39:51.624968 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:51.624975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:51.626931 (+ 1956us) server_negotiation.cc:407] Connection header received
0619 12:39:51.627080 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:51.627084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:51.627128 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:51.627238 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:51.628113 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:51.628865 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:51.630051 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:51.630407 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:51.630523 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:51.632381 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:51.632384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:51.632387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:51.632414 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:51.634234 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:51.634751 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:51.634755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:51.634757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:51.634850 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:51.635257 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:51.635260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:51.635262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:51.635502 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:51.635647 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:51.635985 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 12:39:51.636081 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:39:51.637383 22082 catalog_manager.cc:1347] Generated new cluster ID: d595b8ff23144ad3a8d7a17b19143f29
I20260619 12:39:51.637451 22082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:51.642953 22082 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:51.644210 22082 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:51.660627 22082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7031c14c13d449e8cc69f505281e2be: Generated new TSK 0
I20260619 12:39:51.661428 22082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:51.665833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 21995
W20260619 12:39:51.671232 21980 connection.cc:535] client connection to 127.0.98.62:39731 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39731): Connection reset by peer (error 104)
2026-06-19T12:39:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:51 dist-test-slave-kdrb krb5kdc[22095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:51 dist-test-slave-kdrb krb5kdc[22095](info): set up 2 sockets
Jun 19 12:39:51 dist-test-slave-kdrb krb5kdc[22095](info): commencing operation
krb5kdc: starting...
W20260619 12:39:52.719816 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:52 dist-test-slave-kdrb krb5kdc[22095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:39:52.851922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36977
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:52.969996 22108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:52.970286 22108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:52.970329 22108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:52.974618 22108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:52.974710 22108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:52.974731 22108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:52.974748 22108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:52.974764 22108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:52.980404 22108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:52.981446 22108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:52.982431 22108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:39:52.991340 22108 server_base.cc:1034] running on GCE node
W20260619 12:39:52.991286 22113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:52.991240 22114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:52.991242 22116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:52.992244 22108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:52.993427 22108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:52.994628 22108 hybrid_clock.cc:648] HybridClock initialized: now 1781872792994595 us; error 53 us; skew 500 ppm
Jun 19 12:39:52 dist-test-slave-kdrb krb5kdc[22095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:52.997567 22108 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:39:52.998553 22108 webserver.cc:460] Webserver started at http://127.0.98.62:36889/ using document root <none> and password file <none>
I20260619 12:39:52.999135 22108 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:52.999245 22108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:52.999560 22108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:53.001653 22108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "a8c0c701833f44f787785de41a74f8ef"
format_stamp: "Formatted at 2026-06-19 12:39:52 on dist-test-slave-kdrb"
server_key: "7cf23b80c5a2c57c69b88d45b67527de"
server_key_iv: "c4afdb3a84a6396e13b941e761151c05"
server_key_version: "encryptionkey@0"
I20260619 12:39:53.002275 22108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "a8c0c701833f44f787785de41a74f8ef"
format_stamp: "Formatted at 2026-06-19 12:39:52 on dist-test-slave-kdrb"
server_key: "7cf23b80c5a2c57c69b88d45b67527de"
server_key_iv: "c4afdb3a84a6396e13b941e761151c05"
server_key_version: "encryptionkey@0"
I20260619 12:39:53.006206 22108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 12:39:53.008836 22123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:53.010116 22108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:39:53.010286 22108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "a8c0c701833f44f787785de41a74f8ef"
format_stamp: "Formatted at 2026-06-19 12:39:52 on dist-test-slave-kdrb"
server_key: "7cf23b80c5a2c57c69b88d45b67527de"
server_key_iv: "c4afdb3a84a6396e13b941e761151c05"
server_key_version: "encryptionkey@0"
I20260619 12:39:53.010416 22108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:53.028038 22108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:53.031567 22108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:53.031803 22108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:53.039031 22108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36977
I20260619 12:39:53.039052 22175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36977 every 8 connection(s)
I20260619 12:39:53.040159 22108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:53.043078 22176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:53.048568 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 22108
I20260619 12:39:53.048719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:53.048987 392 external_mini_cluster.cc:1428] Setting key 56d811aaef88ef564392a76f9c5f0df4
I20260619 12:39:53.049149 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef: Bootstrap starting.
I20260619 12:39:53.051753 22176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:53.052664 22176 log.cc:824] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:53.054867 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef: No bootstrap required, opened a new log
I20260619 12:39:53.058488 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:53.050370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37267 (local address 127.0.98.62:36977)
0619 12:39:53.050826 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:53.050846 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:53.050884 (+ 38us) server_negotiation.cc:407] Connection header received
0619 12:39:53.051582 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:53.051602 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:53.052029 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:53.052328 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:53.052699 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:53.053733 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:53.054991 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:53.055534 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:53.055653 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:53.056605 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:53.056610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:53.056624 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:53.057024 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:53.057079 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:53.057103 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:53.057206 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:53.057557 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 12:39:53.057736 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":158,"threads_started":1}
I20260619 12:39:53.058995 22176 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c0c701833f44f787785de41a74f8ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36977 } }
I20260619 12:39:53.059217 22176 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:53.059252 22176 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c0c701833f44f787785de41a74f8ef, State: Initialized, Role: FOLLOWER
I20260619 12:39:53.059830 22176 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c0c701833f44f787785de41a74f8ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36977 } }
I20260619 12:39:53.059991 22176 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:53.060047 22176 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:53.060142 22176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:53.061251 22176 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c0c701833f44f787785de41a74f8ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36977 } }
I20260619 12:39:53.061633 22176 leader_election.cc:302] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8c0c701833f44f787785de41a74f8ef; no voters:
I20260619 12:39:53.061851 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:53.062032 22176 leader_election.cc:288] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:53.062160 22181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:53.062427 22181 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [term 1 LEADER]: Becoming Leader. State: Replica: a8c0c701833f44f787785de41a74f8ef, State: Running, Role: LEADER
I20260619 12:39:53.062790 22181 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c0c701833f44f787785de41a74f8ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36977 } }
I20260619 12:39:53.063457 22176 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:53.064620 22183 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8c0c701833f44f787785de41a74f8ef. Latest consensus state: current_term: 1 leader_uuid: "a8c0c701833f44f787785de41a74f8ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c0c701833f44f787785de41a74f8ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36977 } } }
I20260619 12:39:53.064764 22183 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:53.065089 22182 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8c0c701833f44f787785de41a74f8ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c0c701833f44f787785de41a74f8ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36977 } } }
I20260619 12:39:53.065174 22182 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:53.066164 22195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:53.069449 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:53.064198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49547 (local address 127.0.98.62:36977)
0619 12:39:53.064314 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:53.064318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:53.064721 (+ 403us) server_negotiation.cc:407] Connection header received
0619 12:39:53.064955 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:53.064958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:53.065017 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:53.065085 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:53.065566 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:53.066364 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:53.067518 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:53.067830 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:53.067941 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:53.068921 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:53.068924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:53.068927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:53.068942 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:53.068973 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:53.068985 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:53.069082 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:53.069261 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 12:39:53.069313 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":45}
I20260619 12:39:53.069553 22195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:53.075335 22195 catalog_manager.cc:1347] Generated new cluster ID: 118a2976b0344988acd90b31f49b4f48
I20260619 12:39:53.075413 22195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:53.085342 22195 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:53.086676 22195 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:53.100786 22195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8c0c701833f44f787785de41a74f8ef: Generated new TSK 0
I20260619 12:39:53.101738 22195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:53.106519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 22108
W20260619 12:39:53.112185 22093 connection.cc:535] client connection to 127.0.98.62:36977 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36977): Connection reset by peer (error 104)
2026-06-19T12:39:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:53 dist-test-slave-kdrb krb5kdc[22207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:53 dist-test-slave-kdrb krb5kdc[22207](info): set up 2 sockets
Jun 19 12:39:53 dist-test-slave-kdrb krb5kdc[22207](info): commencing operation
krb5kdc: starting...
W20260619 12:39:54.175886 392 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:54 dist-test-slave-kdrb krb5kdc[22207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:39:54.302402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40027
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:54.418830 22220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:54.419207 22220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:54.419305 22220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:54.423527 22220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:54.423671 22220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:54.423713 22220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:54.423755 22220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:54.423799 22220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:54.429657 22220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40027
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:54.431129 22220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:54.432271 22220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:54.441490 22226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:54.441512 22228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:54.441605 22220 server_base.cc:1034] running on GCE node
W20260619 12:39:54.441612 22225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:54.442453 22220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:54.443738 22220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:54.444938 22220 hybrid_clock.cc:648] HybridClock initialized: now 1781872794444924 us; error 29 us; skew 500 ppm
Jun 19 12:39:54 dist-test-slave-kdrb krb5kdc[22207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:54.447888 22220 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:39:54.448787 22220 webserver.cc:460] Webserver started at http://127.0.98.62:44917/ using document root <none> and password file <none>
I20260619 12:39:54.449323 22220 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:54.449393 22220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:54.449566 22220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:54.451803 22220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "09a70664e71d41d88da93092d1d6de1b"
format_stamp: "Formatted at 2026-06-19 12:39:54 on dist-test-slave-kdrb"
server_key: "d8916b5efbd932f07143617166630df4"
server_key_iv: "535762d5fe55eead552cdea167b4a4ba"
server_key_version: "encryptionkey@0"
I20260619 12:39:54.452405 22220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "09a70664e71d41d88da93092d1d6de1b"
format_stamp: "Formatted at 2026-06-19 12:39:54 on dist-test-slave-kdrb"
server_key: "d8916b5efbd932f07143617166630df4"
server_key_iv: "535762d5fe55eead552cdea167b4a4ba"
server_key_version: "encryptionkey@0"
I20260619 12:39:54.456244 22220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 12:39:54.458766 22235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:54.460119 22220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 12:39:54.460278 22220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "09a70664e71d41d88da93092d1d6de1b"
format_stamp: "Formatted at 2026-06-19 12:39:54 on dist-test-slave-kdrb"
server_key: "d8916b5efbd932f07143617166630df4"
server_key_iv: "535762d5fe55eead552cdea167b4a4ba"
server_key_version: "encryptionkey@0"
I20260619 12:39:54.460407 22220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:54.465188 22220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:54.468272 22220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:54.468498 22220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:54.475600 22220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40027
I20260619 12:39:54.475631 22287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40027 every 8 connection(s)
I20260619 12:39:54.476572 22220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:54.479061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 22220
I20260619 12:39:54.479197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:54.479552 392 external_mini_cluster.cc:1428] Setting key f2bb4174d1f318da5b694b5b4c4927de
I20260619 12:39:54.479672 22288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:54.485270 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:54.481040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45593 (local address 127.0.98.62:40027)
0619 12:39:54.481509 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:54.481526 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:54.481563 (+ 37us) server_negotiation.cc:407] Connection header received
0619 12:39:54.482485 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:54.482504 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:54.482799 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:54.483156 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:54.484034 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:54.484044 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:54.484057 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:54.484067 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:54.484122 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:54.484142 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:54.484302 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:54.484385 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 12:39:54.484551 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":146,"threads_started":1}
I20260619 12:39:54.486761 22288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b: Bootstrap starting.
I20260619 12:39:54.488225 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:54.489385 22288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:54.490231 22288 log.cc:824] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:54.492262 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:54.490357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45651 (local address 127.0.98.62:40027)
0619 12:39:54.490481 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:54.490485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:54.490746 (+ 261us) server_negotiation.cc:407] Connection header received
0619 12:39:54.490834 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:54.490836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:54.490882 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:54.490974 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:54.491809 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:54.491812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:54.491814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:54.491824 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:54.491864 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:54.491875 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:54.491945 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:54.492066 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 12:39:54.492100 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 12:39:54.492743 22288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b: No bootstrap required, opened a new log
I20260619 12:39:54.496469 22288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a70664e71d41d88da93092d1d6de1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40027 } }
I20260619 12:39:54.496709 22288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:54.496802 22288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a70664e71d41d88da93092d1d6de1b, State: Initialized, Role: FOLLOWER
I20260619 12:39:54.497304 22288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a70664e71d41d88da93092d1d6de1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40027 } }
I20260619 12:39:54.497468 22288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:54.497539 22288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:54.497644 22288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:54.498632 22288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a70664e71d41d88da93092d1d6de1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40027 } }
I20260619 12:39:54.499028 22288 leader_election.cc:302] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09a70664e71d41d88da93092d1d6de1b; no voters:
I20260619 12:39:54.499432 22288 leader_election.cc:288] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:54.499608 22298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:54.499850 22298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [term 1 LEADER]: Becoming Leader. State: Replica: 09a70664e71d41d88da93092d1d6de1b, State: Running, Role: LEADER
I20260619 12:39:54.500202 22298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a70664e71d41d88da93092d1d6de1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40027 } }
I20260619 12:39:54.500798 22288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:54.502039 22300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09a70664e71d41d88da93092d1d6de1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a70664e71d41d88da93092d1d6de1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40027 } } }
I20260619 12:39:54.502009 22299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09a70664e71d41d88da93092d1d6de1b. Latest consensus state: current_term: 1 leader_uuid: "09a70664e71d41d88da93092d1d6de1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a70664e71d41d88da93092d1d6de1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40027 } } }
I20260619 12:39:54.502216 22299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:54.502247 22300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:54.502620 22308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:54.506127 22308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:54.511368 22308 catalog_manager.cc:1347] Generated new cluster ID: e856bba7e7f14f15a38c671951f97f77
I20260619 12:39:54.511448 22308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:54.522071 22308 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:54.522398 22308 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:54.530629 22308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09a70664e71d41d88da93092d1d6de1b: Generated new TSK 0
I20260619 12:39:54.531494 22308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:54.539405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 22220
2026-06-19T12:39:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:54 dist-test-slave-kdrb krb5kdc[22319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:54 dist-test-slave-kdrb krb5kdc[22319](info): set up 2 sockets
Jun 19 12:39:54 dist-test-slave-kdrb krb5kdc[22319](info): commencing operation
krb5kdc: starting...
W20260619 12:39:55.575857 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:55 dist-test-slave-kdrb krb5kdc[22319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:39:55.703028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43451
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:55.816803 22332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:55.817143 22332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:55.817224 22332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:55.821353 22332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:55.821476 22332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:55.821516 22332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:55.821554 22332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:55.821589 22332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:55.827041 22332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:55.828198 22332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:55.829272 22332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:55.838069 22338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:55.838091 22340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:55.838091 22337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:55.838637 22332 server_base.cc:1034] running on GCE node
I20260619 12:39:55.839259 22332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:55.840494 22332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:55.841668 22332 hybrid_clock.cc:648] HybridClock initialized: now 1781872795841642 us; error 42 us; skew 500 ppm
Jun 19 12:39:55 dist-test-slave-kdrb krb5kdc[22319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:55.844592 22332 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:39:55.845541 22332 webserver.cc:460] Webserver started at http://127.0.98.62:39705/ using document root <none> and password file <none>
I20260619 12:39:55.846066 22332 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:55.846135 22332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:55.846307 22332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:55.848419 22332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "ea3a11ab56f741afa3648a5a1a25c447"
format_stamp: "Formatted at 2026-06-19 12:39:55 on dist-test-slave-kdrb"
server_key: "50bdf8827bdc59a6f3d01877822f3967"
server_key_iv: "a48189a87b4e8593d4077a6afcf303b9"
server_key_version: "encryptionkey@0"
I20260619 12:39:55.848999 22332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea3a11ab56f741afa3648a5a1a25c447"
format_stamp: "Formatted at 2026-06-19 12:39:55 on dist-test-slave-kdrb"
server_key: "50bdf8827bdc59a6f3d01877822f3967"
server_key_iv: "a48189a87b4e8593d4077a6afcf303b9"
server_key_version: "encryptionkey@0"
I20260619 12:39:55.852958 22332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:39:55.855592 22347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:55.856922 22332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:39:55.857096 22332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "ea3a11ab56f741afa3648a5a1a25c447"
format_stamp: "Formatted at 2026-06-19 12:39:55 on dist-test-slave-kdrb"
server_key: "50bdf8827bdc59a6f3d01877822f3967"
server_key_iv: "a48189a87b4e8593d4077a6afcf303b9"
server_key_version: "encryptionkey@0"
I20260619 12:39:55.857235 22332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:55.898433 22332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:55.901491 22332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:55.901712 22332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:55.909186 22332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43451
I20260619 12:39:55.909206 22399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43451 every 8 connection(s)
I20260619 12:39:55.910175 22332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:55.913276 22400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:55.918905 22400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447: Bootstrap starting.
I20260619 12:39:55.920100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 22332
I20260619 12:39:55.920262 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:55.920612 392 external_mini_cluster.cc:1428] Setting key 7a97d2a851f6738cd9fa325da805134d
I20260619 12:39:55.921803 22400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:55.922564 22400 log.cc:824] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:55.925011 22400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447: No bootstrap required, opened a new log
I20260619 12:39:55.929482 22400 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3a11ab56f741afa3648a5a1a25c447" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43451 } }
I20260619 12:39:55.929827 22400 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:55.929901 22400 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3a11ab56f741afa3648a5a1a25c447, State: Initialized, Role: FOLLOWER
I20260619 12:39:55.930359 22400 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3a11ab56f741afa3648a5a1a25c447" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43451 } }
I20260619 12:39:55.930477 22400 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Jun 19 12:39:55 dist-test-slave-kdrb krb5kdc[22319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:39:55.930517 22400 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:55.930636 22400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:55.931910 22400 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3a11ab56f741afa3648a5a1a25c447" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43451 } }
I20260619 12:39:55.932387 22400 leader_election.cc:302] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea3a11ab56f741afa3648a5a1a25c447; no voters:
I20260619 12:39:55.932793 22400 leader_election.cc:288] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:55.932919 22405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:55.933132 22405 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [term 1 LEADER]: Becoming Leader. State: Replica: ea3a11ab56f741afa3648a5a1a25c447, State: Running, Role: LEADER
I20260619 12:39:55.933490 22405 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3a11ab56f741afa3648a5a1a25c447" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43451 } }
I20260619 12:39:55.934212 22400 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:55.934918 22407 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea3a11ab56f741afa3648a5a1a25c447. Latest consensus state: current_term: 1 leader_uuid: "ea3a11ab56f741afa3648a5a1a25c447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3a11ab56f741afa3648a5a1a25c447" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43451 } } }
I20260619 12:39:55.935050 22407 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:55.935451 22411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:55.935436 22406 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea3a11ab56f741afa3648a5a1a25c447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3a11ab56f741afa3648a5a1a25c447" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43451 } } }
I20260619 12:39:55.936702 22406 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:55.937894 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:55.921973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54877 (local address 127.0.98.62:43451)
0619 12:39:55.922457 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:55.922480 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:55.922517 (+ 37us) server_negotiation.cc:407] Connection header received
0619 12:39:55.923456 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:55.923546 (+ 90us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:55.923934 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:55.924228 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:55.925185 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:55.926235 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:55.927535 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:55.928031 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:55.928154 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:55.931243 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:55.931249 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:55.931264 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:55.931296 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:55.933650 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:55.934248 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:55.934254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:55.934260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:55.934407 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:55.934763 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:55.934769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:55.934773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:55.935397 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:55.935622 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:55.936556 (+ 934us) server_negotiation.cc:299] Negotiation successful
0619 12:39:55.936886 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":151,"threads_started":1}
I20260619 12:39:55.939184 22411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:55.941525 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:55.949466 22411 catalog_manager.cc:1347] Generated new cluster ID: 5959bf06d14148bc951716f07b3a4efb
I20260619 12:39:55.949540 22411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:55.952941 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:55.944209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43801 (local address 127.0.98.62:43451)
0619 12:39:55.944345 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:55.944349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:55.944365 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:39:55.944430 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:55.944433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:55.944483 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:55.944570 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:55.945255 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:55.946068 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:55.946970 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:55.947343 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:55.947456 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:55.949308 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:55.949312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:55.949315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:39:55.949338 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:55.951165 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:55.951690 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:55.951694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:55.951696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:55.951782 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:39:55.952075 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:39:55.952078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:39:55.952080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:39:55.952289 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:55.952459 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:55.952660 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 12:39:55.952774 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:39:55.968286 22411 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:55.969621 22411 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:55.988190 22411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea3a11ab56f741afa3648a5a1a25c447: Generated new TSK 0
I20260619 12:39:55.989147 22411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:55.998193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 22332
2026-06-19T12:39:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:56 dist-test-slave-kdrb krb5kdc[22431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:56 dist-test-slave-kdrb krb5kdc[22431](info): set up 2 sockets
Jun 19 12:39:56 dist-test-slave-kdrb krb5kdc[22431](info): commencing operation
krb5kdc: starting...
W20260619 12:39:57.043833 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:57 dist-test-slave-kdrb krb5kdc[22431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:39:57.172801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37521
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:57.287200 22444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:57.287549 22444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:57.287657 22444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:57.291618 22444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:57.291736 22444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:57.291782 22444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:57.291815 22444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:57.291841 22444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:57.297149 22444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:57.298281 22444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:57.299324 22444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:57.308090 22450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:57.308095 22452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:57.308090 22449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:57.308178 22444 server_base.cc:1034] running on GCE node
I20260619 12:39:57.308974 22444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:57.310098 22444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:57.311318 22444 hybrid_clock.cc:648] HybridClock initialized: now 1781872797311294 us; error 46 us; skew 500 ppm
Jun 19 12:39:57 dist-test-slave-kdrb krb5kdc[22431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:57.318763 22444 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:39:57.319854 22444 webserver.cc:460] Webserver started at http://127.0.98.62:40711/ using document root <none> and password file <none>
I20260619 12:39:57.320457 22444 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:57.320560 22444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:57.320811 22444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:57.322935 22444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "215eaa2542fe48f78da085b6dc56836d"
format_stamp: "Formatted at 2026-06-19 12:39:57 on dist-test-slave-kdrb"
server_key: "531319ee394b8cb2bbf7ef310b2cbe9e"
server_key_iv: "19c8b8f4421afe50afcb149896a3c772"
server_key_version: "encryptionkey@0"
I20260619 12:39:57.323648 22444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "215eaa2542fe48f78da085b6dc56836d"
format_stamp: "Formatted at 2026-06-19 12:39:57 on dist-test-slave-kdrb"
server_key: "531319ee394b8cb2bbf7ef310b2cbe9e"
server_key_iv: "19c8b8f4421afe50afcb149896a3c772"
server_key_version: "encryptionkey@0"
I20260619 12:39:57.327879 22444 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 12:39:57.330322 22459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:57.331612 22444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:39:57.331943 22444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "215eaa2542fe48f78da085b6dc56836d"
format_stamp: "Formatted at 2026-06-19 12:39:57 on dist-test-slave-kdrb"
server_key: "531319ee394b8cb2bbf7ef310b2cbe9e"
server_key_iv: "19c8b8f4421afe50afcb149896a3c772"
server_key_version: "encryptionkey@0"
I20260619 12:39:57.332082 22444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:57.345907 22444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:57.349184 22444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:57.349427 22444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:57.357580 22444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37521
I20260619 12:39:57.357592 22511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37521 every 8 connection(s)
I20260619 12:39:57.358618 22444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:57.359396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 22444
I20260619 12:39:57.359668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:57.360040 392 external_mini_cluster.cc:1428] Setting key 793933c41361a69891ddc51b210694b4
I20260619 12:39:57.362270 22512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:57.369848 22512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d: Bootstrap starting.
I20260619 12:39:57.370374 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:57.361671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50027 (local address 127.0.98.62:37521)
0619 12:39:57.362084 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:57.362102 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:57.362147 (+ 45us) server_negotiation.cc:407] Connection header received
0619 12:39:57.362835 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:57.362852 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:57.363191 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:57.363558 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:57.363887 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:57.364918 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:57.366298 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:57.366804 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:57.366942 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:57.368371 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:57.368379 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:57.368397 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:57.368961 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:57.369023 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:57.369050 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:57.369170 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:57.369295 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 12:39:57.369537 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":151,"threads_started":1}
I20260619 12:39:57.372630 22512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:57.373584 22512 log.cc:824] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:57.373629 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:57.376313 22512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d: No bootstrap required, opened a new log
I20260619 12:39:57.380239 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:57.375842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54765 (local address 127.0.98.62:37521)
0619 12:39:57.375981 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:57.375985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:57.376196 (+ 211us) server_negotiation.cc:407] Connection header received
0619 12:39:57.376316 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:57.376318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:57.376359 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:57.376461 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:57.376792 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:57.377520 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:57.378312 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:39:57.378751 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:39:57.378874 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:39:57.379745 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:57.379748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:57.379751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:57.379764 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:57.379791 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:57.379804 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:57.379886 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:57.380035 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 12:39:57.380093 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:39:57.381179 22512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215eaa2542fe48f78da085b6dc56836d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37521 } }
I20260619 12:39:57.381415 22512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:57.381485 22512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 215eaa2542fe48f78da085b6dc56836d, State: Initialized, Role: FOLLOWER
I20260619 12:39:57.381934 22512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215eaa2542fe48f78da085b6dc56836d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37521 } }
I20260619 12:39:57.382056 22512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:57.382139 22512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:57.382248 22512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:57.383226 22512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215eaa2542fe48f78da085b6dc56836d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37521 } }
I20260619 12:39:57.383661 22512 leader_election.cc:302] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 215eaa2542fe48f78da085b6dc56836d; no voters:
I20260619 12:39:57.384003 22512 leader_election.cc:288] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:57.384305 22522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:57.384526 22522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [term 1 LEADER]: Becoming Leader. State: Replica: 215eaa2542fe48f78da085b6dc56836d, State: Running, Role: LEADER
I20260619 12:39:57.384892 22522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215eaa2542fe48f78da085b6dc56836d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37521 } }
I20260619 12:39:57.385334 22512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:57.386544 22523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "215eaa2542fe48f78da085b6dc56836d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215eaa2542fe48f78da085b6dc56836d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37521 } } }
I20260619 12:39:57.386698 22523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:57.387048 22524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 215eaa2542fe48f78da085b6dc56836d. Latest consensus state: current_term: 1 leader_uuid: "215eaa2542fe48f78da085b6dc56836d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215eaa2542fe48f78da085b6dc56836d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37521 } } }
I20260619 12:39:57.387146 22524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:57.387156 22531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:57.390499 22531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:57.396526 22531 catalog_manager.cc:1347] Generated new cluster ID: 2f4c093e74514d5b80751ffd2b925609
I20260619 12:39:57.396653 22531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:57.414971 22531 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:57.416267 22531 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:57.423419 22531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 215eaa2542fe48f78da085b6dc56836d: Generated new TSK 0
I20260619 12:39:57.424393 22531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:57.435772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 22444
2026-06-19T12:39:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:57 dist-test-slave-kdrb krb5kdc[22543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:57 dist-test-slave-kdrb krb5kdc[22543](info): set up 2 sockets
Jun 19 12:39:57 dist-test-slave-kdrb krb5kdc[22543](info): commencing operation
krb5kdc: starting...
W20260619 12:39:58.479897 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:58 dist-test-slave-kdrb krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:39:58.606564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39689
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:39:58.720109 22556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:39:58.720453 22556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:39:58.720534 22556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:39:58.724714 22556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:39:58.724828 22556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:39:58.724874 22556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:39:58.724933 22556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:39:58.724972 22556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:39:58.730492 22556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:39:58.731686 22556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:39:58.732784 22556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:39:58.741415 22561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:39:58.741420 22564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:58.741637 22556 server_base.cc:1034] running on GCE node
W20260619 12:39:58.741434 22562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:39:58.742380 22556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:39:58.743599 22556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:39:58.744809 22556 hybrid_clock.cc:648] HybridClock initialized: now 1781872798744795 us; error 25 us; skew 500 ppm
Jun 19 12:39:58 dist-test-slave-kdrb krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:39:58.747579 22556 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:39:58.748571 22556 webserver.cc:460] Webserver started at http://127.0.98.62:43659/ using document root <none> and password file <none>
I20260619 12:39:58.749143 22556 fs_manager.cc:359] Metadata directory not provided
I20260619 12:39:58.749238 22556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:39:58.749478 22556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:39:58.751621 22556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "7c0f5b468f314220b0292a8058ee9d39"
format_stamp: "Formatted at 2026-06-19 12:39:58 on dist-test-slave-kdrb"
server_key: "11356dd47a95e688bbca95f1b4fe83ef"
server_key_iv: "62ec53a6b929e54c82d6abe49bef405e"
server_key_version: "encryptionkey@0"
I20260619 12:39:58.752206 22556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "7c0f5b468f314220b0292a8058ee9d39"
format_stamp: "Formatted at 2026-06-19 12:39:58 on dist-test-slave-kdrb"
server_key: "11356dd47a95e688bbca95f1b4fe83ef"
server_key_iv: "62ec53a6b929e54c82d6abe49bef405e"
server_key_version: "encryptionkey@0"
I20260619 12:39:58.756170 22556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:39:58.758662 22571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:39:58.759982 22556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:39:58.760141 22556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "7c0f5b468f314220b0292a8058ee9d39"
format_stamp: "Formatted at 2026-06-19 12:39:58 on dist-test-slave-kdrb"
server_key: "11356dd47a95e688bbca95f1b4fe83ef"
server_key_iv: "62ec53a6b929e54c82d6abe49bef405e"
server_key_version: "encryptionkey@0"
I20260619 12:39:58.760264 22556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:39:58.764935 22556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:39:58.767884 22556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:39:58.768086 22556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:39:58.775292 22556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39689
I20260619 12:39:58.775323 22623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39689 every 8 connection(s)
I20260619 12:39:58.776325 22556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:39:58.779060 22624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:39:58.783104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 22556
I20260619 12:39:58.783218 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:39:58.783567 392 external_mini_cluster.cc:1428] Setting key 3b1f47fe50bfcca291e0bfdb9ed4a9c5
I20260619 12:39:58.785000 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39: Bootstrap starting.
I20260619 12:39:58.787395 22624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39: Neither blocks nor log segments found. Creating new log.
I20260619 12:39:58.788269 22624 log.cc:824] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39: Log is configured to *not* fsync() on all Append() calls
I20260619 12:39:58.789098 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:58.785016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59757 (local address 127.0.98.62:39689)
0619 12:39:58.785473 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:58.785488 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:58.785524 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:39:58.786170 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:58.786189 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:58.786479 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:58.786847 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:58.787662 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:58.787667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:58.787680 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:58.787690 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:58.787744 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:58.787764 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:58.787894 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:58.788008 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 12:39:58.788168 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":147,"threads_started":1}
I20260619 12:39:58.790501 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39: No bootstrap required, opened a new log
I20260619 12:39:58.792068 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:39:58.794593 22624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0f5b468f314220b0292a8058ee9d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } }
I20260619 12:39:58.794859 22624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:39:58.794924 22624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0f5b468f314220b0292a8058ee9d39, State: Initialized, Role: FOLLOWER
I20260619 12:39:58.795497 22624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0f5b468f314220b0292a8058ee9d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } }
I20260619 12:39:58.795718 22624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:39:58.795797 22624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:39:58.795904 22624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:39:58.796051 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:39:58.794143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52123 (local address 127.0.98.62:39689)
0619 12:39:58.794280 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 12:39:58.794295 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:39:58.794518 (+ 223us) server_negotiation.cc:407] Connection header received
0619 12:39:58.794672 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:39:58.794675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:39:58.794718 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:39:58.794813 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:39:58.795606 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:39:58.795609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:39:58.795611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:39:58.795621 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:39:58.795670 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:39:58.795689 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:39:58.795779 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:39:58.795860 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 12:39:58.795901 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 12:39:58.797043 22624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0f5b468f314220b0292a8058ee9d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } }
I20260619 12:39:58.797415 22624 leader_election.cc:302] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c0f5b468f314220b0292a8058ee9d39; no voters:
I20260619 12:39:58.797750 22624 leader_election.cc:288] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:39:58.797820 22634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:39:58.798080 22634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [term 1 LEADER]: Becoming Leader. State: Replica: 7c0f5b468f314220b0292a8058ee9d39, State: Running, Role: LEADER
I20260619 12:39:58.798417 22634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0f5b468f314220b0292a8058ee9d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } }
I20260619 12:39:58.799088 22624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:39:58.800240 22636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c0f5b468f314220b0292a8058ee9d39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0f5b468f314220b0292a8058ee9d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } } }
I20260619 12:39:58.800405 22636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [sys.catalog]: This master's current role is: LEADER
I20260619 12:39:58.800240 22635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c0f5b468f314220b0292a8058ee9d39. Latest consensus state: current_term: 1 leader_uuid: "7c0f5b468f314220b0292a8058ee9d39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0f5b468f314220b0292a8058ee9d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } } }
I20260619 12:39:58.800693 22635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39 [sys.catalog]: This master's current role is: LEADER
W20260619 12:39:58.803766 22649 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:39:58.803840 22649 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:39:58.803920 22643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:39:58.804662 22643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:39:58.810165 22643 catalog_manager.cc:1347] Generated new cluster ID: 02c19f1bbd334a05ab33e464ed1265da
I20260619 12:39:58.810241 22643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:39:58.839171 22643 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:39:58.839604 22643 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:39:58.846853 22643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c0f5b468f314220b0292a8058ee9d39: Generated new TSK 0
I20260619 12:39:58.847668 22643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:39:58.858785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 22556
2026-06-19T12:39:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1423 ms)
[----------] 18 tests from AuthTokenIssuingTest (25709 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:39:58 dist-test-slave-kdrb krb5kdc[22655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:39:58 dist-test-slave-kdrb krb5kdc[22655](info): set up 2 sockets
Jun 19 12:39:58 dist-test-slave-kdrb krb5kdc[22655](info): commencing operation
krb5kdc: starting...
W20260619 12:39:59.903952 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:39:59 dist-test-slave-kdrb krb5kdc[22655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:39:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:39:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:40:00.030669 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34609
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34609,127.0.98.61:40861,127.0.98.60:43815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:40:00.141106 22668 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:40:00.141465 22668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:40:00.141546 22668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:40:00.141625 22668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:40:00.145592 22668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:40:00.145694 22668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:40:00.145735 22668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:40:00.145769 22668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:40:00.145797 22668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:40:00.151108 22668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34609,127.0.98.61:40861,127.0.98.60:43815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:40:00.152388 22668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:40:00.153491 22668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:40:00.162312 22668 server_base.cc:1034] running on GCE node
W20260619 12:40:00.162259 22674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:40:00.162259 22676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:40:00.162256 22673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:40:00.163188 22668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:40:00.164415 22668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:40:00.165621 22668 hybrid_clock.cc:648] HybridClock initialized: now 1781872800165607 us; error 29 us; skew 500 ppm
Jun 19 12:40:00 dist-test-slave-kdrb krb5kdc[22655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:40:00.168557 22668 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:40:00.169488 22668 webserver.cc:460] Webserver started at http://127.0.98.62:39607/ using document root <none> and password file <none>
I20260619 12:40:00.170090 22668 fs_manager.cc:359] Metadata directory not provided
I20260619 12:40:00.170189 22668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:40:00.170439 22668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:40:00.172657 22668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "24dc9d388b84481e82070f18967cd2ab"
format_stamp: "Formatted at 2026-06-19 12:40:00 on dist-test-slave-kdrb"
server_key: "b65c60aea551c8900b0a9b9077692ee3"
server_key_iv: "1b3c1cbfb423fdf9ce97d34732c0019e"
server_key_version: "encryptionkey@0"
I20260619 12:40:00.173259 22668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "24dc9d388b84481e82070f18967cd2ab"
format_stamp: "Formatted at 2026-06-19 12:40:00 on dist-test-slave-kdrb"
server_key: "b65c60aea551c8900b0a9b9077692ee3"
server_key_iv: "1b3c1cbfb423fdf9ce97d34732c0019e"
server_key_version: "encryptionkey@0"
I20260619 12:40:00.177265 22668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:40:00.179879 22683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:40:00.181211 22668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:40:00.181393 22668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "24dc9d388b84481e82070f18967cd2ab"
format_stamp: "Formatted at 2026-06-19 12:40:00 on dist-test-slave-kdrb"
server_key: "b65c60aea551c8900b0a9b9077692ee3"
server_key_iv: "1b3c1cbfb423fdf9ce97d34732c0019e"
server_key_version: "encryptionkey@0"
I20260619 12:40:00.181533 22668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:40:00.218576 22668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:40:00.221822 22668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:40:00.222067 22668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:40:00.229377 22668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34609
I20260619 12:40:00.229671 22735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34609 every 8 connection(s)
I20260619 12:40:00.230387 22668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:40:00.233237 22736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:40:00.237211 22736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:40:00.237658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 22668
I20260619 12:40:00.237794 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:40:00.238425 392 external_mini_cluster.cc:1428] Setting key 9c764a848f7be2ba2120b1ba5d4304c9
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 12:40:00 dist-test-slave-kdrb krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:40:00.256740 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.240166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34609 (local address 127.0.0.1:34788)
0619 12:40:00.240782 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.240845 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:00.241978 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:00.242708 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:00.242717 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:00.243252 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:00.243657 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.243678 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.245561 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.245563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.246270 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.246278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.247196 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.247199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.247288 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.247951 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:00.247966 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:00.251396 (+ 3430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:00.253496 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.253507 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.253519 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.253760 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.254097 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.254102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.254105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.254227 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.254868 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:00.254874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:00.255018 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 12:40:00.255194 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 12:40:00.255649 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":163,"spinlock_wait_cycles":77056,"thread_start_us":146,"threads_started":1}
I20260619 12:40:00.256742 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.240166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34788 (local address 127.0.98.62:34609)
0619 12:40:00.240667 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.240689 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.240973 (+ 284us) server_negotiation.cc:407] Connection header received
0619 12:40:00.242298 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.242303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.242490 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.242646 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.243889 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.245422 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.246399 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.247040 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.247230 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.251585 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:00.251591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:00.251605 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:00.251632 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:00.253370 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.253890 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.253896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.253897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.254002 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.254329 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.254332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.254334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.254734 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:00.254880 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:00.255267 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 12:40:00.255593 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":315,"thread_start_us":146,"threads_started":1}
I20260619 12:40:00.259346 22736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:40:00.260190 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.259919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40861 (local address 127.0.0.1:58018)
0619 12:40:00.260044 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.260120 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 12:40:00.260470 22684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40861: connect: Connection refused (error 111)
I20260619 12:40:00.261353 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.261145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40861 (local address 127.0.0.1:58028)
0619 12:40:00.261242 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.261287 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 12:40:00.261514 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40861: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40861: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 12:40:00.266372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40861
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34609,127.0.98.61:40861,127.0.98.60:43815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:40:00.310820 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } attempt: 1
I20260619 12:40:00.311784 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.311422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40861 (local address 127.0.0.1:58038)
0619 12:40:00.311609 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.311689 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 12:40:00.312575 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.312324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40861 (local address 127.0.0.1:58054)
0619 12:40:00.312457 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.312512 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 12:40:00.312736 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40861: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40861: connect: Connection refused (error 111)
W20260619 12:40:00.380198 22743 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:40:00.380523 22743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:40:00.380548 22743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:40:00.380587 22743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 12:40:00.380916 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } attempt: 2
I20260619 12:40:00.381884 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.381542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40861 (local address 127.0.0.1:58064)
0619 12:40:00.381720 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.381792 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 12:40:00.382645 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.382427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40861 (local address 127.0.0.1:58066)
0619 12:40:00.382530 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.382579 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 12:40:00.382802 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40861: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40861: connect: Connection refused (error 111)
W20260619 12:40:00.385490 22743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:40:00.385552 22743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:40:00.385573 22743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:40:00.385617 22743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:40:00.385653 22743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:40:00.391368 22743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34609,127.0.98.61:40861,127.0.98.60:43815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:40:00.392660 22743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:40:00.393744 22743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:40:00.402575 22749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:40:00.402596 22748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:40:00.402575 22751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:40:00.402876 22743 server_base.cc:1034] running on GCE node
I20260619 12:40:00.403594 22743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:40:00.404685 22743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:40:00.405846 22743 hybrid_clock.cc:648] HybridClock initialized: now 1781872800405832 us; error 28 us; skew 500 ppm
Jun 19 12:40:00 dist-test-slave-kdrb krb5kdc[22655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:40:00.408663 22743 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 12:40:00.409516 22743 webserver.cc:460] Webserver started at http://127.0.98.61:39593/ using document root <none> and password file <none>
I20260619 12:40:00.410058 22743 fs_manager.cc:359] Metadata directory not provided
I20260619 12:40:00.410115 22743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:40:00.410281 22743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:40:00.412456 22743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/data/instance:
uuid: "cfe0f004fbd44976b779ee9f1f2b72ca"
format_stamp: "Formatted at 2026-06-19 12:40:00 on dist-test-slave-kdrb"
server_key: "7127d3af6c18e0a5215f5e0bbb8c6941"
server_key_iv: "4775da79cdb0f007f7b79bbde49f4e77"
server_key_version: "encryptionkey@0"
I20260619 12:40:00.412982 22743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/wal/instance:
uuid: "cfe0f004fbd44976b779ee9f1f2b72ca"
format_stamp: "Formatted at 2026-06-19 12:40:00 on dist-test-slave-kdrb"
server_key: "7127d3af6c18e0a5215f5e0bbb8c6941"
server_key_iv: "4775da79cdb0f007f7b79bbde49f4e77"
server_key_version: "encryptionkey@0"
I20260619 12:40:00.416709 22743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:40:00.419140 22758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:40:00.420441 22743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:40:00.420547 22743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/wal
uuid: "cfe0f004fbd44976b779ee9f1f2b72ca"
format_stamp: "Formatted at 2026-06-19 12:40:00 on dist-test-slave-kdrb"
server_key: "7127d3af6c18e0a5215f5e0bbb8c6941"
server_key_iv: "4775da79cdb0f007f7b79bbde49f4e77"
server_key_version: "encryptionkey@0"
I20260619 12:40:00.420635 22743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:40:00.445137 22743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:40:00.448637 22743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:40:00.448799 22743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:40:00.455893 22743 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40861
I20260619 12:40:00.456189 22810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40861 every 8 connection(s)
I20260619 12:40:00.456913 22743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/data/info.pb
I20260619 12:40:00.459833 22811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:40:00.462895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 22743
I20260619 12:40:00.462992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-1/wal/instance
I20260619 12:40:00.463289 392 external_mini_cluster.cc:1428] Setting key 5b0df9854632ca8f0b75742191a6436b
I20260619 12:40:00.463757 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 12:40:00 dist-test-slave-kdrb krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:40:00.474009 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } attempt: 3
I20260619 12:40:00.478976 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.465957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34800 (local address 127.0.98.62:34609)
0619 12:40:00.466123 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.466128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.467001 (+ 873us) server_negotiation.cc:407] Connection header received
0619 12:40:00.468103 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.468107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.468178 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.468275 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.469513 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.470045 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.471264 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.471612 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.471726 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.474423 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:00.474429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:00.474436 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:00.474462 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:00.476449 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.476974 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.476980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.476982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.477067 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.477429 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.477432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.477434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.477681 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:00.477775 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:00.478522 (+ 747us) server_negotiation.cc:299] Negotiation successful
0619 12:40:00.478707 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 12:40:00 dist-test-slave-kdrb krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 12:40:00.480886 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.466335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34609 (local address 127.0.0.1:34800)
0619 12:40:00.466823 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.466892 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:00.467873 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:00.468451 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:00.468458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:00.468879 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:00.469356 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.469370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.470185 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.470193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.471117 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.471126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.471772 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.471775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.471932 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.472549 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:00.472566 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:00.474228 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:00.476566 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.476572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.476587 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.476865 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.477163 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.477167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.477169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.477320 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.477816 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:00.477825 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:00.478188 (+ 363us) client_negotiation.cc:769] Sending connection context
0619 12:40:00.479244 (+ 1056us) client_negotiation.cc:240] Negotiation successful
0619 12:40:00.479685 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":144,"threads_started":1}
I20260619 12:40:00.483112 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:40:00.485620 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.474488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40861 (local address 127.0.0.1:58082)
0619 12:40:00.474644 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.474661 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:00.475025 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:00.475811 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:00.475816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:00.476048 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:00.476182 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.476193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.477753 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.477756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.478308 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.478315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.479026 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.479030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.479140 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.479690 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:00.479702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:00.480925 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:00.483686 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.483689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.483691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.484033 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.484350 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.484354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.484356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.484448 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.484992 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:00.484996 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:00.485102 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 12:40:00.485276 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 12:40:00.485440 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 12:40:00.485719 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.474814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58082 (local address 127.0.98.61:40861)
0619 12:40:00.475304 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.475319 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.475349 (+ 30us) server_negotiation.cc:407] Connection header received
0619 12:40:00.475399 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.475406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.475652 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.475816 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.476324 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.477586 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.478436 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.478899 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.479023 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.481440 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:00.481446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:00.481456 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:00.481478 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:00.483564 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.484152 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.484158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.484160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.484250 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.484551 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.484554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.484556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.484851 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:00.485017 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:00.485375 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 12:40:00.485521 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":106,"threads_started":1}
I20260619 12:40:00.487589 22736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:40:00.488374 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.488121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43815 (local address 127.0.0.1:60114)
0619 12:40:00.488241 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.488296 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 12:40:00.489342 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.488979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43815 (local address 127.0.0.1:60130)
0619 12:40:00.489164 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.489239 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260619 12:40:00.489516 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43815: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
Jun 19 12:40:00 dist-test-slave-kdrb krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 12:40:00.491528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43815
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34609,127.0.98.61:40861,127.0.98.60:43815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:40:00.494395 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.483720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40861 (local address 127.0.0.1:58090)
0619 12:40:00.483853 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.483867 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:00.483945 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:00.484395 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:00.484398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:00.484566 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:00.484645 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.484650 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.486653 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.486656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.487367 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.487377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.488477 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.488481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.488594 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.489127 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:00.489142 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:00.490604 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:00.492867 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.492870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.492872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.493062 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.493370 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.493375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.493377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.493461 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.493848 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:00.493859 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:00.493949 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 12:40:00.494060 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 12:40:00.494210 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 12:40:00.494441 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.483880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58090 (local address 127.0.98.61:40861)
0619 12:40:00.484202 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.484208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.484218 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:40:00.484244 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.484247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.484279 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.484387 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.484773 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.485579 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.487517 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.488334 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.488422 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.490741 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:00.490744 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:00.490749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:00.490775 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:00.492736 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.493175 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.493181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.493184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.493269 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.493557 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.493560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.493562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.493744 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:00.493851 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:00.494104 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 12:40:00.494273 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":258,"thread_start_us":149,"threads_started":1}
I20260619 12:40:00.495154 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:40:00.496031 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.495666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43815 (local address 127.0.0.1:60138)
0619 12:40:00.495829 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.495939 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 12:40:00.496295 22761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
I20260619 12:40:00.497262 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.496973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43815 (local address 127.0.0.1:60148)
0619 12:40:00.497117 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.497181 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 12:40:00.497421 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43815: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
I20260619 12:40:00.520689 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } attempt: 1
I20260619 12:40:00.521631 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.521316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43815 (local address 127.0.0.1:60160)
0619 12:40:00.521479 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.521546 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 12:40:00.522421 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.522220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43815 (local address 127.0.0.1:60162)
0619 12:40:00.522323 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.522369 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 12:40:00.522575 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43815: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
I20260619 12:40:00.546715 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } attempt: 1
I20260619 12:40:00.547676 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.547348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43815 (local address 127.0.0.1:60168)
0619 12:40:00.547520 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.547592 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 12:40:00.548527 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.548283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43815 (local address 127.0.0.1:60178)
0619 12:40:00.548412 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.548465 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 12:40:00.548702 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43815: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
I20260619 12:40:00.597756 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } attempt: 2
I20260619 12:40:00.598707 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.598351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43815 (local address 127.0.0.1:60180)
0619 12:40:00.598539 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.598612 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 12:40:00.599570 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.599303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43815 (local address 127.0.0.1:60190)
0619 12:40:00.599408 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.599452 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 12:40:00.599695 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43815: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
W20260619 12:40:00.611296 22818 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:40:00.611706 22818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:40:00.611781 22818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:40:00.611858 22818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:40:00.616047 22818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:40:00.616140 22818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:40:00.616179 22818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:40:00.616217 22818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:40:00.616253 22818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:40:00.616957 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } attempt: 2
I20260619 12:40:00.617827 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.617528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43815 (local address 127.0.0.1:60192)
0619 12:40:00.617675 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.617749 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 12:40:00.618538 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.618322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43815 (local address 127.0.0.1:60204)
0619 12:40:00.618426 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.618477 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 12:40:00.618656 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43815: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43815: connect: Connection refused (error 111)
I20260619 12:40:00.622128 22818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34609,127.0.98.61:40861,127.0.98.60:43815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:40:00.623302 22818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:40:00.624421 22818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:40:00.633409 22824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:40:00.633392 22825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:40:00.633594 22818 server_base.cc:1034] running on GCE node
W20260619 12:40:00.633392 22827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:40:00.634326 22818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:40:00.635562 22818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:40:00.636766 22818 hybrid_clock.cc:648] HybridClock initialized: now 1781872800636750 us; error 30 us; skew 500 ppm
Jun 19 12:40:00 dist-test-slave-kdrb krb5kdc[22655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:40:00.639555 22818 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 12:40:00.640470 22818 webserver.cc:460] Webserver started at http://127.0.98.60:45105/ using document root <none> and password file <none>
I20260619 12:40:00.641047 22818 fs_manager.cc:359] Metadata directory not provided
I20260619 12:40:00.641140 22818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:40:00.641371 22818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:40:00.643405 22818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/data/instance:
uuid: "c13b4b3046064a058372ecccf05e08b8"
format_stamp: "Formatted at 2026-06-19 12:40:00 on dist-test-slave-kdrb"
server_key: "8b35dcd12730acb88e46f2a182e3a404"
server_key_iv: "70008f3ca34925afe9a6b6880f9b570b"
server_key_version: "encryptionkey@0"
I20260619 12:40:00.644037 22818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/wal/instance:
uuid: "c13b4b3046064a058372ecccf05e08b8"
format_stamp: "Formatted at 2026-06-19 12:40:00 on dist-test-slave-kdrb"
server_key: "8b35dcd12730acb88e46f2a182e3a404"
server_key_iv: "70008f3ca34925afe9a6b6880f9b570b"
server_key_version: "encryptionkey@0"
I20260619 12:40:00.647756 22818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:40:00.650259 22834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:40:00.651615 22818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:40:00.651774 22818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/wal
uuid: "c13b4b3046064a058372ecccf05e08b8"
format_stamp: "Formatted at 2026-06-19 12:40:00 on dist-test-slave-kdrb"
server_key: "8b35dcd12730acb88e46f2a182e3a404"
server_key_iv: "70008f3ca34925afe9a6b6880f9b570b"
server_key_version: "encryptionkey@0"
I20260619 12:40:00.651897 22818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:40:00.662520 22818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:40:00.665642 22818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:40:00.665872 22818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:40:00.672981 22818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43815
I20260619 12:40:00.673272 22886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43815 every 8 connection(s)
I20260619 12:40:00.674041 22818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/data/info.pb
I20260619 12:40:00.677076 22887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:40:00.678622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 22818
I20260619 12:40:00.678747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872516972220-392-0/minicluster-data/master-2/wal/instance
I20260619 12:40:00.679064 392 external_mini_cluster.cc:1428] Setting key a11ff6fb0d1a8692a46cd88ba8c98e2e
I20260619 12:40:00.682132 22887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:40:00 dist-test-slave-kdrb krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 12:40:00 dist-test-slave-kdrb krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:40:00.693877 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.684218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34814 (local address 127.0.98.62:34609)
0619 12:40:00.684384 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.684389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.684756 (+ 367us) server_negotiation.cc:407] Connection header received
0619 12:40:00.684913 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.684920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.684996 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.685078 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.685895 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.686521 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.687751 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.687973 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.688088 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.690532 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:00.690536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:00.690539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:00.690575 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:00.691956 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.692507 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.692511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.692513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.692596 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.692889 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.692893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.692895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.693084 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:00.693192 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:00.693565 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 12:40:00.693697 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 12:40:00.694734 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.684152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34609 (local address 127.0.0.1:34814)
0619 12:40:00.684617 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.684667 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:00.684790 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:00.685170 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:00.685175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:00.685646 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:00.685767 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.685773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.686635 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.686638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.687633 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.687646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.688130 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.688134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.688248 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.688799 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:00.688816 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:00.690388 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:00.692071 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.692082 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.692092 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.692397 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.692694 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.692698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.692700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.692787 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.693191 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:00.693199 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:00.693342 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 12:40:00.693500 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 12:40:00.693877 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":108,"spinlock_wait_cycles":3840,"thread_start_us":169,"threads_started":1}
I20260619 12:40:00.694741 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.680495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.0.98.60:43815)
0619 12:40:00.680941 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.680962 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.681008 (+ 46us) server_negotiation.cc:407] Connection header received
0619 12:40:00.681881 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.681907 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.682392 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.682788 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.683550 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.685006 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.686013 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.686631 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.686793 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.689599 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:00.689605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:00.689617 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:00.689638 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:00.691619 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.692074 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.692081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.692088 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.692241 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.692546 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.692550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.692552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.692974 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:00.693180 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:00.693451 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 12:40:00.693789 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"spinlock_wait_cycles":6400,"thread_start_us":113,"threads_started":1}
I20260619 12:40:00.695968 22887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:40:00.698868 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } attempt: 3
Jun 19 12:40:00 dist-test-slave-kdrb krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 12:40:00 dist-test-slave-kdrb krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 12:40:00 dist-test-slave-kdrb krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 12:40:00.705603 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.696445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40861 (local address 127.0.0.1:58104)
0619 12:40:00.696564 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.696591 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:00.696672 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:00.697023 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:00.697029 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:00.697224 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:00.697352 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.697360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.698164 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.698169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.698695 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.698702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.699510 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.699514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.699621 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.700157 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:00.700172 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:00.701813 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:00.703872 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.703876 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.703880 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.704149 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.704474 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.704478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.704481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.704582 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.705005 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:00.705008 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:00.705108 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 12:40:00.705229 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 12:40:00.705382 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"spinlock_wait_cycles":3712}
I20260619 12:40:00.705837 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.696584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58104 (local address 127.0.98.61:40861)
0619 12:40:00.696739 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.696743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.696757 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:40:00.696803 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.696806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.696866 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.696955 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.697496 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.698044 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.698819 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.699159 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.699297 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.701960 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:00.701963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:00.701965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:00.701992 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:00.703746 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.704267 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.704274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.704279 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.704377 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.704675 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.704681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.704682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.704898 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:00.705017 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:00.705295 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 12:40:00.705667 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 12:40:00.706840 22887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:40:00.707598 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.697258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.0.98.62:34609)
0619 12:40:00.697385 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.697389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.697402 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:40:00.697506 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.697509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.697547 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.697634 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.698249 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.698737 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.699715 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.700036 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.700140 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.703458 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:00.703462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:00.703465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:00.703525 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:00.705344 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.705848 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.705851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.705853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.705943 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.706258 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.706262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.706264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.706440 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:00.706701 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:00.707242 (+ 541us) server_negotiation.cc:299] Negotiation successful
0619 12:40:00.707412 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 12:40:00.709178 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.699263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43815 (local address 127.0.0.1:60230)
0619 12:40:00.699553 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.699567 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:00.699636 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:00.700421 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:00.700424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:00.700588 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:00.700709 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.700719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.701458 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.701463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.702240 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.702251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.702788 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.702792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.702870 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.703320 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:00.703334 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:00.704928 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:00.707220 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.707225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.707227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.707529 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.708081 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.708084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.708087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.708184 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.708675 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:00.708681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:00.708784 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 12:40:00.708881 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 12:40:00.709018 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226}
I20260619 12:40:00.709798 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } attempt: 3
I20260619 12:40:00.710285 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.699422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.0.98.60:43815)
0619 12:40:00.699578 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.699583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.699598 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:40:00.700241 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.700247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.700291 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.700827 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.700904 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.701347 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.702403 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.702660 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.702777 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.705085 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:00.705090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:00.705093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:00.705122 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:00.707113 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.707883 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.707886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.707887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.707968 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.708288 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.708291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.708292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.708554 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:00.709651 (+ 1097us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:00.709725 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 12:40:00.709825 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":86,"spinlock_wait_cycles":768}
Jun 19 12:40:00 dist-test-slave-kdrb krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 12:40:00.714636 22736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab: Bootstrap starting.
Jun 19 12:40:00 dist-test-slave-kdrb krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 12:40:00 dist-test-slave-kdrb krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 12:40:00.717787 22736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab: Neither blocks nor log segments found. Creating new log.
I20260619 12:40:00.717937 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.707402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43815 (local address 127.0.0.1:60246)
0619 12:40:00.707626 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.707642 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:00.707722 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:00.708231 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:00.708237 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:00.708418 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:00.708531 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.708537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.710063 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.710069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.710939 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.710949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.711687 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.711694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.711760 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.712266 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:00.712295 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:00.713989 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:00.716261 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.716264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.716266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.716546 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.716794 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.716796 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.716798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.716869 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.717313 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:00.717315 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:00.717390 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 12:40:00.717611 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 12:40:00.717773 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":2,"spinlock_wait_cycles":3456}
I20260619 12:40:00.718405 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.707633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60246 (local address 127.0.98.60:43815)
0619 12:40:00.707991 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.708003 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.708016 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:40:00.708050 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.708055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.708100 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.708183 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.708646 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.709966 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.711097 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.711446 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.711581 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.714376 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:00.714379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:00.714382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:00.714406 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:00.716145 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.716640 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.716643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.716645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.716714 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.716947 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.716949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.716951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.717183 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:00.717270 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:00.718135 (+ 865us) server_negotiation.cc:299] Negotiation successful
0619 12:40:00.718237 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":138,"threads_started":1}
I20260619 12:40:00.719679 22736 log.cc:824] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab: Log is configured to *not* fsync() on all Append() calls
I20260619 12:40:00.720371 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.710162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43815 (local address 127.0.0.1:60256)
0619 12:40:00.710271 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:00.710287 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:00.710373 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:00.710730 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:00.710733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:00.711077 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:00.711188 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.711196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.712399 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.712405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.713174 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:00.713186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.713811 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.713815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:00.713917 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.714369 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:00.714388 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:00.716090 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:00.718729 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.718732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.718736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.718985 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.719317 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:00.719320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:00.719322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:00.719419 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:00.719913 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:00.719916 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:00.719999 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 12:40:00.720073 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 12:40:00.720208 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 12:40:00.720817 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.708980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58110 (local address 127.0.98.61:40861)
0619 12:40:00.709092 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.709097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.709109 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:40:00.709144 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.709146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.709190 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.709282 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.710510 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.711315 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.712471 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.712799 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.712868 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.716306 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:00.716309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:00.716311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:00.716335 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:00.718394 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.718912 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.718917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.718921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.719022 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.719331 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.719334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.719336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.719741 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:00.719966 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:00.720288 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 12:40:00.720391 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:40:00.720958 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.710227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.0.98.60:43815)
0619 12:40:00.710517 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.710525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.710536 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:40:00.710566 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.710569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.710606 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.711309 (+ 703us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.711447 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.712240 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.713317 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.713670 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.714157 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.716261 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:00.716265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:00.716267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:00.716308 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:00.718613 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.719117 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.719120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.719122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.719212 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.719583 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.719589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.719593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.719800 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:00.720634 (+ 834us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:00.720708 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 12:40:00.720810 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"spinlock_wait_cycles":9472,"thread_start_us":112,"threads_started":1}
I20260619 12:40:00.721624 22887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8: Bootstrap starting.
I20260619 12:40:00.723693 22811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca: Bootstrap starting.
I20260619 12:40:00.724084 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:40:00.724385 22887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8: Neither blocks nor log segments found. Creating new log.
I20260619 12:40:00.725329 22887 log.cc:824] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8: Log is configured to *not* fsync() on all Append() calls
I20260619 12:40:00.726401 22811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca: Neither blocks nor log segments found. Creating new log.
I20260619 12:40:00.725618 22710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "24dc9d388b84481e82070f18967cd2ab"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34826
I20260619 12:40:00.727210 22811 log.cc:824] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca: Log is configured to *not* fsync() on all Append() calls
I20260619 12:40:00.727596 22887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8: No bootstrap required, opened a new log
I20260619 12:40:00.727618 22736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab: No bootstrap required, opened a new log
I20260619 12:40:00.730285 22811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca: No bootstrap required, opened a new log
I20260619 12:40:00.730692 22736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } }
I20260619 12:40:00.730958 22736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24dc9d388b84481e82070f18967cd2ab, State: Initialized, Role: FOLLOWER
I20260619 12:40:00.731314 22736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } }
I20260619 12:40:00.731405 22887 raft_consensus.cc:348] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } }
I20260619 12:40:00.731734 22887 raft_consensus.cc:729] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13b4b3046064a058372ecccf05e08b8, State: Initialized, Role: FOLLOWER
I20260619 12:40:00.731963 22897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } } }
I20260619 12:40:00.732064 22887 consensus_queue.cc:260] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } }
I20260619 12:40:00.732218 22897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:40:00.732828 22736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:40:00.732899 22710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 12:40:00.732995 22710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:40:00.733210 22887 sys_catalog.cc:564] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:40:00.734424 22710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } }
I20260619 12:40:00.735340 22898 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } } }
I20260619 12:40:00.736958 22710 leader_election.cc:288] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [CANDIDATE]: Term 1 election: Requested vote from peers cfe0f004fbd44976b779ee9f1f2b72ca (127.0.98.61:40861), c13b4b3046064a058372ecccf05e08b8 (127.0.98.60:43815)
I20260619 12:40:00.735781 22898 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:40:00.737637 22861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24dc9d388b84481e82070f18967cd2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c13b4b3046064a058372ecccf05e08b8"
I20260619 12:40:00.737910 22861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8 [term 0 FOLLOWER]: Advancing to term 1
W20260619 12:40:00.738478 22916 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:40:00.738539 22916 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 12:40:00.738663 22920 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:40:00.738727 22920 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:40:00.739138 22861 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24dc9d388b84481e82070f18967cd2ab in term 1.
I20260619 12:40:00.739627 22686 leader_election.cc:302] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24dc9d388b84481e82070f18967cd2ab, c13b4b3046064a058372ecccf05e08b8; no voters:
I20260619 12:40:00.739914 22897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:40:00.740121 22897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [term 1 LEADER]: Becoming Leader. State: Replica: 24dc9d388b84481e82070f18967cd2ab, State: Running, Role: LEADER
I20260619 12:40:00.740188 22811 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } }
I20260619 12:40:00.740398 22811 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfe0f004fbd44976b779ee9f1f2b72ca, State: Initialized, Role: FOLLOWER
I20260619 12:40:00.740473 22897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } }
I20260619 12:40:00.740718 22811 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } }
I20260619 12:40:00.742061 22811 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:40:00.742106 22785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24dc9d388b84481e82070f18967cd2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca"
I20260619 12:40:00.742269 22897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24dc9d388b84481e82070f18967cd2ab. Latest consensus state: current_term: 1 leader_uuid: "24dc9d388b84481e82070f18967cd2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } } }
I20260619 12:40:00.742322 22785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:40:00.742357 22897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [sys.catalog]: This master's current role is: LEADER
I20260619 12:40:00.742745 22931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:40:00.743687 22785 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24dc9d388b84481e82070f18967cd2ab in term 1.
I20260619 12:40:00.743716 22931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:40:00.743616 22668 master_runner.cc:186] Error getting master registration for 127.0.98.61:40861: OK, instance_id { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" instance_seqno: 1781872800453932 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:40:00.747009 22668 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:40:00.750988 22818 master_runner.cc:263] No leader master found from master c13b4b3046064a058372ecccf05e08b8
I20260619 12:40:00.751057 22818 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:40:00.751654 22861 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8 [term 1 FOLLOWER]: Refusing update from remote peer 24dc9d388b84481e82070f18967cd2ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:40:00.752399 22897 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:40:00.754949 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.741110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34842 (local address 127.0.98.62:34609)
0619 12:40:00.741232 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.741238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.745589 (+ 4351us) server_negotiation.cc:407] Connection header received
0619 12:40:00.745783 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.745786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.745831 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.745882 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.746897 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.747665 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.748842 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.749151 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.749228 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.751196 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:00.751199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:00.751202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:00.751229 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:00.753120 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.753608 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.753612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.753614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.753694 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.753981 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.753985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.753987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.754241 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:00.754376 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:00.754626 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 12:40:00.754776 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:40:00.757019 22898 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24dc9d388b84481e82070f18967cd2ab. Latest consensus state: current_term: 1 leader_uuid: "24dc9d388b84481e82070f18967cd2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } } }
I20260619 12:40:00.757175 22898 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:40:00.757558 22785 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca [term 1 FOLLOWER]: Refusing update from remote peer 24dc9d388b84481e82070f18967cd2ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:40:00.758147 22897 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:40:00.761807 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.742183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.0.98.61:40861)
0619 12:40:00.743963 (+ 1780us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.743968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.751450 (+ 7482us) server_negotiation.cc:407] Connection header received
0619 12:40:00.751666 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.751672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.751737 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.751858 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.753078 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.753853 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.755460 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.755818 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.755900 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.757742 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:00.757746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:00.757749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:00.757777 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:00.759555 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.760062 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.760066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.760068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.760151 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.760474 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.760477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.760479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.761019 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:00.761128 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:00.761521 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 12:40:00.761640 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1709}
I20260619 12:40:00.763916 22948 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24dc9d388b84481e82070f18967cd2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } } }
I20260619 12:40:00.764061 22948 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:40:00.764164 22948 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24dc9d388b84481e82070f18967cd2ab. Latest consensus state: current_term: 1 leader_uuid: "24dc9d388b84481e82070f18967cd2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } } }
I20260619 12:40:00.764242 22948 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:40:00.766516 22897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24dc9d388b84481e82070f18967cd2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } } }
I20260619 12:40:00.766640 22897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [sys.catalog]: This master's current role is: LEADER
I20260619 12:40:00.767765 22898 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24dc9d388b84481e82070f18967cd2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } } }
I20260619 12:40:00.767890 22898 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 12:40:00.769099 22942 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:40:00.769183 22942 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:40:00.770107 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:00.747196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.0.98.60:43815)
0619 12:40:00.750434 (+ 3238us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.750438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.756232 (+ 5794us) server_negotiation.cc:407] Connection header received
0619 12:40:00.756274 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.756276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.756327 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.756407 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.757901 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.758664 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.759721 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:00.760045 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:00.760117 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:00.763451 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:00.763456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:00.763458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:00.763505 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:00.765247 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.768682 (+ 3435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.768686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.768688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.768779 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:00.769083 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:00.769086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:00.769088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:00.769338 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:00.769459 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:00.769816 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 12:40:00.769956 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3157}
I20260619 12:40:00.770696 22945 mvcc.cc:205] Tried to move back new op lower bound from 7298550991871315968 to 7298550991836487680. Current Snapshot: MvccSnapshot[applied={T|T < 7298550991871315968}]
I20260619 12:40:00.771032 22946 mvcc.cc:205] Tried to move back new op lower bound from 7298550991871315968 to 7298550991836487680. Current Snapshot: MvccSnapshot[applied={T|T < 7298550991871315968}]
I20260619 12:40:00.773321 22897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24dc9d388b84481e82070f18967cd2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } } }
I20260619 12:40:00.773459 22897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab [sys.catalog]: This master's current role is: LEADER
I20260619 12:40:00.774755 22928 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24dc9d388b84481e82070f18967cd2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24dc9d388b84481e82070f18967cd2ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } peers { permanent_uuid: "cfe0f004fbd44976b779ee9f1f2b72ca" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40861 } } peers { permanent_uuid: "c13b4b3046064a058372ecccf05e08b8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43815 } } }
I20260619 12:40:00.774894 22928 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:40:00.775557 22949 mvcc.cc:205] Tried to move back new op lower bound from 7298550991871315968 to 7298550991836487680. Current Snapshot: MvccSnapshot[applied={T|T < 7298550991871315968}]
I20260619 12:40:00.776739 22931 catalog_manager.cc:1347] Generated new cluster ID: 51a5a370350d4c3dbee1ab49d9b8b8a1
I20260619 12:40:00.776809 22931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:40:00.790778 22931 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:40:00.792225 22931 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:40:00.805334 22931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24dc9d388b84481e82070f18967cd2ab: Generated new TSK 0
I20260619 12:40:00.806488 22931 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:40:00.836135 22739 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:00.833292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.0.98.62:34609)
0619 12:40:00.833433 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.833438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.834867 (+ 1429us) server_negotiation.cc:407] Connection header received
0619 12:40:00.835102 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.835105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.835165 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.835222 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.835981 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34850: BlockingRecv error: recv got EOF from 127.0.0.1:34850 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 12:40:00.838200 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:00.834186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.0.98.60:43815)
0619 12:40:00.834303 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.834308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.836885 (+ 2577us) server_negotiation.cc:407] Connection header received
0619 12:40:00.837335 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.837339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.837392 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.837445 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.838056 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60272: BlockingRecv error: recv got EOF from 127.0.0.1:60272 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 12:40:00.840287 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:00.838891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58124 (local address 127.0.98.61:40861)
0619 12:40:00.839009 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.839014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.839265 (+ 251us) server_negotiation.cc:407] Connection header received
0619 12:40:00.839441 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.839444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.839516 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.839569 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.840153 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58124: BlockingRecv error: recv got EOF from 127.0.0.1:58124 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 12:40:00.845773 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:00.843912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58138 (local address 127.0.98.61:40861)
0619 12:40:00.844035 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.844040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.844640 (+ 600us) server_negotiation.cc:407] Connection header received
0619 12:40:00.844912 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.844916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.845001 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.845058 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.845636 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58138: BlockingRecv error: recv got EOF from 127.0.0.1:58138 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 12:40:00.851408 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:00.849827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.0.98.61:40861)
0619 12:40:00.849936 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.849940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.850437 (+ 497us) server_negotiation.cc:407] Connection header received
0619 12:40:00.850630 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.850634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.850689 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.850755 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.851290 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58150: BlockingRecv error: recv got EOF from 127.0.0.1:58150 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 12:40:00.857923 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:00.856157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.0.98.61:40861)
0619 12:40:00.856271 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.856275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.856878 (+ 603us) server_negotiation.cc:407] Connection header received
0619 12:40:00.857079 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.857083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.857137 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.857190 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.857768 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58166: BlockingRecv error: recv got EOF from 127.0.0.1:58166 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 12:40:00.866353 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:00.864648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.0.98.61:40861)
0619 12:40:00.864807 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.864811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.865339 (+ 528us) server_negotiation.cc:407] Connection header received
0619 12:40:00.865557 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.865561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.865619 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.865673 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.866219 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58182: BlockingRecv error: recv got EOF from 127.0.0.1:58182 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260619 12:40:00.879132 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:00.877354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.0.98.61:40861)
0619 12:40:00.877485 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.877490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.878068 (+ 578us) server_negotiation.cc:407] Connection header received
0619 12:40:00.878284 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.878288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.878357 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.878413 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.879008 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58188: BlockingRecv error: recv got EOF from 127.0.0.1:58188 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 12:40:00.900107 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:00.898137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.0.98.61:40861)
0619 12:40:00.898271 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.898275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.898954 (+ 679us) server_negotiation.cc:407] Connection header received
0619 12:40:00.899193 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.899198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.899263 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.899321 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.899978 (+ 657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58194: BlockingRecv error: recv got EOF from 127.0.0.1:58194 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 12:40:00.936931 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:00.935080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58198 (local address 127.0.98.61:40861)
0619 12:40:00.935226 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:00.935230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:00.935893 (+ 663us) server_negotiation.cc:407] Connection header received
0619 12:40:00.936104 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:00.936108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:00.936176 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:00.936231 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:00.936809 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58198: BlockingRecv error: recv got EOF from 127.0.0.1:58198 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 12:40:01.005169 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:01.003864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.0.98.61:40861)
0619 12:40:01.004016 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:01.004022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:01.004188 (+ 166us) server_negotiation.cc:407] Connection header received
0619 12:40:01.004354 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:01.004356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:01.004406 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:01.004471 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:01.005041 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58214: BlockingRecv error: recv got EOF from 127.0.0.1:58214 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 12:40:01.136619 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:01.135532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58224 (local address 127.0.98.61:40861)
0619 12:40:01.135670 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:01.135674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:01.135852 (+ 178us) server_negotiation.cc:407] Connection header received
0619 12:40:01.135968 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:01.135971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:01.136020 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:01.136103 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:01.136521 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58224: BlockingRecv error: recv got EOF from 127.0.0.1:58224 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 12:40:01.396335 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:01.395022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.0.98.61:40861)
0619 12:40:01.395189 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:01.395194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:01.395318 (+ 124us) server_negotiation.cc:407] Connection header received
0619 12:40:01.395562 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:01.395566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:01.395624 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:01.395723 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:01.396214 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58240: BlockingRecv error: recv got EOF from 127.0.0.1:58240 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 12:40:01.740034 22920 catalog_manager.cc:1259] Loaded cluster ID: 51a5a370350d4c3dbee1ab49d9b8b8a1
I20260619 12:40:01.740165 22920 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8: loading cluster ID for follower catalog manager: success
I20260619 12:40:01.741858 22920 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8: acquiring CA information for follower catalog manager: success
I20260619 12:40:01.743335 22920 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c13b4b3046064a058372ecccf05e08b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:40:01.770614 22942 catalog_manager.cc:1259] Loaded cluster ID: 51a5a370350d4c3dbee1ab49d9b8b8a1
I20260619 12:40:01.770747 22942 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca: loading cluster ID for follower catalog manager: success
I20260619 12:40:01.772553 22942 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca: acquiring CA information for follower catalog manager: success
I20260619 12:40:01.773869 22942 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cfe0f004fbd44976b779ee9f1f2b72ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:40:01.914522 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:01.910662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58254 (local address 127.0.98.61:40861)
0619 12:40:01.910897 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:01.910901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:01.910967 (+ 66us) server_negotiation.cc:407] Connection header received
0619 12:40:01.911094 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:01.911097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:01.911220 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:01.911332 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:40:01.911681 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:01.912221 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:01.913102 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:01.913325 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:01.913415 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:01.913705 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:40:01.914149 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:40:01.914244 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:01.914331 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 12:40:01.914393 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":86,"threads_started":1}
I20260619 12:40:01.921702 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:01.917921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60280 (local address 127.0.98.60:43815)
0619 12:40:01.918167 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:01.918171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:01.918185 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:40:01.918263 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:01.918266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:01.918387 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:01.918489 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:40:01.918687 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:01.919208 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:01.920089 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:01.920312 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:01.920395 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:01.920789 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:40:01.921241 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:40:01.921346 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:01.921472 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 12:40:01.921545 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260619 12:40:01.929152 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:01.924982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58264 (local address 127.0.98.61:40861)
0619 12:40:01.925113 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:01.925120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:01.925369 (+ 249us) server_negotiation.cc:407] Connection header received
0619 12:40:01.925615 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:01.925620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:01.925682 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:01.925761 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:40:01.926042 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:01.926662 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:01.927989 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:01.928215 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:01.928356 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:01.928625 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:40:01.928702 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:40:01.928786 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:01.928930 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 12:40:01.928995 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:40:01.929258 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:01.925065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.0.98.60:43815)
0619 12:40:01.925196 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:01.925200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:01.925543 (+ 343us) server_negotiation.cc:407] Connection header received
0619 12:40:01.925633 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:01.925638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:01.925690 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:01.925784 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:40:01.926105 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:01.926815 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:01.927995 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:01.928330 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:01.928440 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:01.928764 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:40:01.928840 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:40:01.928929 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:01.929072 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 12:40:01.929123 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:40:01.930564 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:01.925011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.0.98.62:34609)
0619 12:40:01.925344 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:01.925397 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:01.926138 (+ 741us) server_negotiation.cc:407] Connection header received
0619 12:40:01.926236 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:01.926239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:01.926415 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:01.926535 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:40:01.926785 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:01.927340 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:01.928836 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:01.929195 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:01.929333 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:01.929751 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:40:01.930143 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:40:01.930261 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:01.930345 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 12:40:01.930405 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":65,"threads_started":1}
I20260619 12:40:01.933966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 22668
I20260619 12:40:01.940578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 22743
I20260619 12:40:01.946309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 22818
2026-06-19T12:40:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3089 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:40:02 dist-test-slave-kdrb krb5kdc[23038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:40:02 dist-test-slave-kdrb krb5kdc[23038](info): set up 2 sockets
Jun 19 12:40:02 dist-test-slave-kdrb krb5kdc[23038](info): commencing operation
krb5kdc: starting...
W20260619 12:40:02.996073 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/test-user.keytab.
Jun 19 12:40:03 dist-test-slave-kdrb krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:40:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:40:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:40:03.123723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39303
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39303,127.0.98.61:35683,127.0.98.60:46145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:40:03.237056 23051 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:40:03.237370 23051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:40:03.237393 23051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:40:03.237485 23051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:40:03.241596 23051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:40:03.241668 23051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:40:03.241688 23051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:40:03.241704 23051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:40:03.241720 23051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:40:03.247167 23051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39303,127.0.98.61:35683,127.0.98.60:46145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:40:03.248494 23051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:40:03.249963 23051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:40:03.259902 23056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:40:03.259783 23059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:40:03.259965 23051 server_base.cc:1034] running on GCE node
W20260619 12:40:03.259783 23057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:40:03.260795 23051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:40:03.262081 23051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:40:03.263296 23051 hybrid_clock.cc:648] HybridClock initialized: now 1781872803263282 us; error 38 us; skew 500 ppm
Jun 19 12:40:03 dist-test-slave-kdrb krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:40:03.272137 23051 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:40:03.273226 23051 webserver.cc:460] Webserver started at http://127.0.98.62:37425/ using document root <none> and password file <none>
I20260619 12:40:03.273860 23051 fs_manager.cc:359] Metadata directory not provided
I20260619 12:40:03.273959 23051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:40:03.274204 23051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:40:03.276533 23051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/data/instance:
uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5"
format_stamp: "Formatted at 2026-06-19 12:40:03 on dist-test-slave-kdrb"
server_key: "925fb063fa38edfdc61cc0f94779d3ad"
server_key_iv: "257fc83f65e5b3821176cd623ee30650"
server_key_version: "encryptionkey@0"
I20260619 12:40:03.277150 23051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/wal/instance:
uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5"
format_stamp: "Formatted at 2026-06-19 12:40:03 on dist-test-slave-kdrb"
server_key: "925fb063fa38edfdc61cc0f94779d3ad"
server_key_iv: "257fc83f65e5b3821176cd623ee30650"
server_key_version: "encryptionkey@0"
I20260619 12:40:03.281140 23051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 12:40:03.283748 23066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:40:03.285034 23051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:40:03.285185 23051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/wal
uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5"
format_stamp: "Formatted at 2026-06-19 12:40:03 on dist-test-slave-kdrb"
server_key: "925fb063fa38edfdc61cc0f94779d3ad"
server_key_iv: "257fc83f65e5b3821176cd623ee30650"
server_key_version: "encryptionkey@0"
I20260619 12:40:03.285311 23051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:40:03.314637 23051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:40:03.318266 23051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:40:03.318486 23051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:40:03.325798 23051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39303
I20260619 12:40:03.326094 23118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39303 every 8 connection(s)
I20260619 12:40:03.326850 23051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:40:03.329733 23119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:40:03.331012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 23051
I20260619 12:40:03.331161 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-0/wal/instance
I20260619 12:40:03.331560 392 external_mini_cluster.cc:1428] Setting key b8759a49d012c7d7ec36ead36d53f987
I20260619 12:40:03.333928 23119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 12:40:03 dist-test-slave-kdrb krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:40:03.351402 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.336854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.0.98.62:39303)
0619 12:40:03.337614 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.337629 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.337683 (+ 54us) server_negotiation.cc:407] Connection header received
0619 12:40:03.338799 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.338807 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.339118 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.339348 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.340435 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.341497 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.342769 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.343162 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.343284 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.346064 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:03.346072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:03.346088 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:03.346118 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:03.347987 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.348627 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.348634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.348635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.348764 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.349205 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.349208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.349209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.349663 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:03.349790 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:03.350325 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 12:40:03.350549 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":615,"spinlock_wait_cycles":42368,"thread_start_us":186,"threads_started":1}
I20260619 12:40:03.351408 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.336852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39303 (local address 127.0.0.1:52948)
0619 12:40:03.337456 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.337519 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:03.338440 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:03.339368 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:03.339373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:03.339814 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:03.340244 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.340261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.341803 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.341810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.342645 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.342655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.343318 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.343322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.343422 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.344193 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:03.344211 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:03.345883 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:03.348115 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.348127 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.348138 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.348489 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.348888 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.348894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.348898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.349072 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.349801 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:03.349808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:03.350007 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 12:40:03.350226 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 12:40:03.350661 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":164,"thread_start_us":149,"threads_started":1}
I20260619 12:40:03.353853 23119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:40:03.354749 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.354390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35683 (local address 127.0.0.1:33148)
0619 12:40:03.354559 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.354656 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 12:40:03.354986 23069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35683: connect: Connection refused (error 111)
I20260619 12:40:03.355979 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.355742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35683 (local address 127.0.0.1:33154)
0619 12:40:03.355841 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.355887 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 12:40:03.356177 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35683: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35683: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 12:40:03.360463 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35683
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39303,127.0.98.61:35683,127.0.98.60:46145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:40:03.405465 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } attempt: 1
I20260619 12:40:03.406454 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.406126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35683 (local address 127.0.0.1:33156)
0619 12:40:03.406290 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.406371 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 12:40:03.407291 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.407023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35683 (local address 127.0.0.1:33160)
0619 12:40:03.407175 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.407227 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 12:40:03.407457 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35683: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35683: connect: Connection refused (error 111)
W20260619 12:40:03.475106 23126 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:40:03.475420 23126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:40:03.475445 23126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:40:03.475564 23126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 12:40:03.475688 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } attempt: 2
I20260619 12:40:03.476503 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.476200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35683 (local address 127.0.0.1:50514)
0619 12:40:03.476353 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.476428 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 12:40:03.477233 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.476999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35683 (local address 127.0.0.1:50530)
0619 12:40:03.477113 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.477162 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 12:40:03.477367 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35683: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35683: connect: Connection refused (error 111)
W20260619 12:40:03.480142 23126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:40:03.480203 23126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:40:03.480224 23126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:40:03.480242 23126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:40:03.480257 23126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:40:03.485805 23126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39303,127.0.98.61:35683,127.0.98.60:46145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:40:03.486954 23126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:40:03.488082 23126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:40:03.497107 23132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:40:03.497077 23131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:40:03.497077 23134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:40:03.497116 23126 server_base.cc:1034] running on GCE node
I20260619 12:40:03.497987 23126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:40:03.499154 23126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:40:03.500344 23126 hybrid_clock.cc:648] HybridClock initialized: now 1781872803500297 us; error 62 us; skew 500 ppm
Jun 19 12:40:03 dist-test-slave-kdrb krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:40:03.503216 23126 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 12:40:03.504287 23126 webserver.cc:460] Webserver started at http://127.0.98.61:34225/ using document root <none> and password file <none>
I20260619 12:40:03.504887 23126 fs_manager.cc:359] Metadata directory not provided
I20260619 12:40:03.504982 23126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:40:03.505223 23126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:40:03.507326 23126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/data/instance:
uuid: "c0f656dbd32545d9beb385545308d5fa"
format_stamp: "Formatted at 2026-06-19 12:40:03 on dist-test-slave-kdrb"
server_key: "2171f8f2762fb49b2aa7b916d1c47f7e"
server_key_iv: "34b39690117634a740d4f6a8aeb9b740"
server_key_version: "encryptionkey@0"
I20260619 12:40:03.507977 23126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/wal/instance:
uuid: "c0f656dbd32545d9beb385545308d5fa"
format_stamp: "Formatted at 2026-06-19 12:40:03 on dist-test-slave-kdrb"
server_key: "2171f8f2762fb49b2aa7b916d1c47f7e"
server_key_iv: "34b39690117634a740d4f6a8aeb9b740"
server_key_version: "encryptionkey@0"
I20260619 12:40:03.511780 23126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 12:40:03.514423 23141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:40:03.515808 23126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:40:03.515954 23126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/wal
uuid: "c0f656dbd32545d9beb385545308d5fa"
format_stamp: "Formatted at 2026-06-19 12:40:03 on dist-test-slave-kdrb"
server_key: "2171f8f2762fb49b2aa7b916d1c47f7e"
server_key_iv: "34b39690117634a740d4f6a8aeb9b740"
server_key_version: "encryptionkey@0"
I20260619 12:40:03.516075 23126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:40:03.532131 23126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:40:03.535276 23126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:40:03.535547 23126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:40:03.542850 23126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35683
I20260619 12:40:03.543219 23193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35683 every 8 connection(s)
I20260619 12:40:03.544560 23126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/data/info.pb
I20260619 12:40:03.547206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 23126
I20260619 12:40:03.547364 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-1/wal/instance
I20260619 12:40:03.547801 392 external_mini_cluster.cc:1428] Setting key 0b5bd2d85c059eb1008d933cfbee5554
I20260619 12:40:03.549218 23194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:40:03.554979 23194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 12:40:03 dist-test-slave-kdrb krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:40:03.568552 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } attempt: 3
I20260619 12:40:03.575196 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.557256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45002 (local address 127.0.98.62:39303)
0619 12:40:03.557403 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.557408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.558696 (+ 1288us) server_negotiation.cc:407] Connection header received
0619 12:40:03.559903 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.559907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.559973 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.560056 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.561812 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.562682 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.564414 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.564858 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.564987 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.568666 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:03.568671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:03.568678 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:03.568715 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:03.571572 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.572300 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.572306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.572308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.572425 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.572925 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.572939 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.572942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.573206 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:03.573298 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:03.574839 (+ 1541us) server_negotiation.cc:299] Negotiation successful
0619 12:40:03.575036 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 12:40:03.575836 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.557786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39303 (local address 127.0.0.1:45002)
0619 12:40:03.558523 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.558580 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:03.559612 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:03.560270 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:03.560280 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:03.560838 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:03.561597 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.561621 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.562827 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.562831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.564235 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.564250 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.565213 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.565223 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.565377 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.566156 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:03.566182 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:03.568402 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:03.571708 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.571718 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.571738 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.572150 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.572556 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.572561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.572565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.572800 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.573372 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:03.573380 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:03.573817 (+ 437us) client_negotiation.cc:769] Sending connection context
0619 12:40:03.574064 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 12:40:03.574521 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":169,"threads_started":1}
Jun 19 12:40:03 dist-test-slave-kdrb krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 12:40:03.577489 23194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:40:03.580452 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.569059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35683 (local address 127.0.0.1:50546)
0619 12:40:03.569198 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.569213 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:03.569314 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:03.570322 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:03.570325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:03.570515 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:03.570649 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.570657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.572275 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.572278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.573180 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.573191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.574105 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.574111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.574231 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.574761 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:03.574780 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:03.576263 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:03.578640 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.578642 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.578644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.578866 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.579154 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.579158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.579160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.579294 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.579884 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:03.579888 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:03.579995 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 12:40:03.580118 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 12:40:03.580277 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 12:40:03.580587 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.569154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.0.98.61:35683)
0619 12:40:03.569807 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.569819 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.569846 (+ 27us) server_negotiation.cc:407] Connection header received
0619 12:40:03.569893 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.569900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.570145 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.570321 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.570767 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.572092 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.573376 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.573960 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.574487 (+ 527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.576402 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:03.576411 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:03.576426 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:03.576455 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:03.578539 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.578966 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.578971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.578972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.579049 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.579397 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.579403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.579404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.579739 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:03.579949 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:03.580217 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 12:40:03.580410 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":172,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 12:40:03.582481 23119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:40:03.583315 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.583013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46145 (local address 127.0.0.1:35894)
0619 12:40:03.583181 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.583240 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 12:40:03.583746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGObR4d/build/debug/bin/kudu
/tmp/dist-test-taskGObR4d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46145
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39303,127.0.98.61:35683,127.0.98.60:46145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:40:03 dist-test-slave-kdrb krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 12:40:03.584645 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.584356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46145 (local address 127.0.0.1:35900)
0619 12:40:03.584507 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.584567 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260619 12:40:03.584785 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
I20260619 12:40:03.588631 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.577950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35683 (local address 127.0.0.1:50560)
0619 12:40:03.578155 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.578173 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:03.578250 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:03.578975 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:03.578978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:03.579099 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:03.579197 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.579202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.580145 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.580151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.580923 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.580937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.581932 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.581935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.582039 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.582550 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:03.582568 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:03.584557 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:03.586765 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.586773 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.586778 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.587087 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.587391 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.587403 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.587408 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.587550 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.587941 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:03.587947 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:03.588075 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 12:40:03.588215 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 12:40:03.588398 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260619 12:40:03.589356 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.578056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.0.98.61:35683)
0619 12:40:03.578425 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.578435 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.578452 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:40:03.578490 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.578495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.578539 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.578629 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.579344 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.580043 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.581046 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.581809 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.581901 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.584734 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:03.584738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:03.584744 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:03.584770 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:03.586594 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.587213 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.587215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.587216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.587293 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.587644 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.587646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.587647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.587835 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:03.587925 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:03.588896 (+ 971us) server_negotiation.cc:299] Negotiation successful
0619 12:40:03.589067 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":127,"threads_started":1}
I20260619 12:40:03.590137 23194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:40:03.590914 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.590649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46145 (local address 127.0.0.1:35910)
0619 12:40:03.590767 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.590847 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 12:40:03.591355 23143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
I20260619 12:40:03.592547 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.592265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46145 (local address 127.0.0.1:35918)
0619 12:40:03.592437 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.592487 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260619 12:40:03.592818 23194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
I20260619 12:40:03.615957 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } attempt: 1
I20260619 12:40:03.617100 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.616762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46145 (local address 127.0.0.1:35922)
0619 12:40:03.616946 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.617017 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 12:40:03.617961 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.617702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46145 (local address 127.0.0.1:35924)
0619 12:40:03.617832 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.617890 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 12:40:03.618086 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
I20260619 12:40:03.642151 23194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } attempt: 1
I20260619 12:40:03.643081 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.642778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46145 (local address 127.0.0.1:35930)
0619 12:40:03.642932 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.643000 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 12:40:03.643844 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.643639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46145 (local address 127.0.0.1:35942)
0619 12:40:03.643743 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.643794 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 12:40:03.643975 23194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
I20260619 12:40:03.693292 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } attempt: 2
I20260619 12:40:03.694211 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.693835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46145 (local address 127.0.0.1:35946)
0619 12:40:03.694036 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.694114 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 12:40:03.695025 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.694776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46145 (local address 127.0.0.1:35960)
0619 12:40:03.694895 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.694958 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 12:40:03.695165 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
W20260619 12:40:03.702957 23201 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:40:03.703271 23201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:40:03.703342 23201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:40:03.703408 23201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:40:03.707633 23201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:40:03.707701 23201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:40:03.707721 23201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:40:03.707778 23201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:40:03.707816 23201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:40:03.712203 23194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } attempt: 2
I20260619 12:40:03.713104 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.712794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46145 (local address 127.0.0.1:35972)
0619 12:40:03.712948 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.713023 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 12:40:03.713934 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.713680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46145 (local address 127.0.0.1:35984)
0619 12:40:03.713815 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.713871 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 12:40:03.714105 23194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46145: connect: Connection refused (error 111)
I20260619 12:40:03.714238 23201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39303,127.0.98.61:35683,127.0.98.60:46145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:40:03.715433 23201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:40:03.716568 23201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:40:03.725548 23208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:40:03.725548 23210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:40:03.725571 23207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:40:03.725790 23201 server_base.cc:1034] running on GCE node
I20260619 12:40:03.726590 23201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:40:03.727849 23201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:40:03.729038 23201 hybrid_clock.cc:648] HybridClock initialized: now 1781872803729024 us; error 30 us; skew 500 ppm
Jun 19 12:40:03 dist-test-slave-kdrb krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:40:03.732432 23201 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 12:40:03.733440 23201 webserver.cc:460] Webserver started at http://127.0.98.60:46799/ using document root <none> and password file <none>
I20260619 12:40:03.734030 23201 fs_manager.cc:359] Metadata directory not provided
I20260619 12:40:03.734097 23201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:40:03.734273 23201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:40:03.736620 23201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/data/instance:
uuid: "5aaeec22e90446bd9b70fc9b8b8682f6"
format_stamp: "Formatted at 2026-06-19 12:40:03 on dist-test-slave-kdrb"
server_key: "df2571d503bf28259a4c7f5a33b0853e"
server_key_iv: "a2b8632679a0846828e6f9fe0b1b7f25"
server_key_version: "encryptionkey@0"
I20260619 12:40:03.737192 23201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/wal/instance:
uuid: "5aaeec22e90446bd9b70fc9b8b8682f6"
format_stamp: "Formatted at 2026-06-19 12:40:03 on dist-test-slave-kdrb"
server_key: "df2571d503bf28259a4c7f5a33b0853e"
server_key_iv: "a2b8632679a0846828e6f9fe0b1b7f25"
server_key_version: "encryptionkey@0"
I20260619 12:40:03.741202 23201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:40:03.743860 23217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:40:03.745206 23201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 12:40:03.745319 23201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/wal
uuid: "5aaeec22e90446bd9b70fc9b8b8682f6"
format_stamp: "Formatted at 2026-06-19 12:40:03 on dist-test-slave-kdrb"
server_key: "df2571d503bf28259a4c7f5a33b0853e"
server_key_iv: "a2b8632679a0846828e6f9fe0b1b7f25"
server_key_version: "encryptionkey@0"
I20260619 12:40:03.745424 23201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:40:03.763851 23201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:40:03.767005 23201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:40:03.767175 23201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:40:03.774452 23201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46145
I20260619 12:40:03.774739 23269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46145 every 8 connection(s)
I20260619 12:40:03.775523 23201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/data/info.pb
I20260619 12:40:03.778331 23270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:40:03.781945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGObR4d/build/debug/bin/kudu as pid 23201
I20260619 12:40:03.782099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGObR4d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872516972220-392-0/minicluster-data/master-2/wal/instance
I20260619 12:40:03.782420 392 external_mini_cluster.cc:1428] Setting key f50f5bff2995020fb0665570199aaf14
I20260619 12:40:03.782294 23270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:40:03 dist-test-slave-kdrb krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 12:40:03 dist-test-slave-kdrb krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 12:40:03.794337 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } attempt: 3
I20260619 12:40:03.795914 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.784519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45004 (local address 127.0.98.62:39303)
0619 12:40:03.784683 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.784688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.784993 (+ 305us) server_negotiation.cc:407] Connection header received
0619 12:40:03.785532 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.785535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.785582 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.785659 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.786695 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.787276 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.788418 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.788629 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.788745 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.791656 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:03.791663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:03.791669 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:03.791709 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:03.793489 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.793989 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.793993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.793995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.794081 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.794438 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.794443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.794445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.794723 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:03.794812 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:03.795581 (+ 769us) server_negotiation.cc:299] Negotiation successful
0619 12:40:03.795719 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 12:40:03.797113 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.783894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.0.98.60:46145)
0619 12:40:03.784313 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.784330 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.784374 (+ 44us) server_negotiation.cc:407] Connection header received
0619 12:40:03.785314 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.785338 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.785594 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.785780 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.786509 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.787517 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.788339 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.788814 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.788940 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.792080 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:03.792085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:03.792097 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:03.792118 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:03.793714 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.794196 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.794202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.794204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.794357 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.794699 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.794701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.794702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.795289 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:03.795519 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:03.795820 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 12:40:03.796135 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":291,"spinlock_wait_cycles":6656,"thread_start_us":142,"threads_started":1}
I20260619 12:40:03.797120 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.784430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39303 (local address 127.0.0.1:45004)
0619 12:40:03.784827 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.784890 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:03.785314 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:03.785739 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:03.785744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:03.786208 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:03.786493 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.786508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.787395 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.787399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.788239 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.788248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.788804 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.788809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.788945 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.789600 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:03.789620 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:03.791342 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:03.793613 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.793624 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.793636 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.793878 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.794175 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.794180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.794182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.794331 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.794878 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:03.794885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:03.795287 (+ 402us) client_negotiation.cc:769] Sending connection context
0619 12:40:03.796059 (+ 772us) client_negotiation.cc:240] Negotiation successful
0619 12:40:03.796272 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":30,"spinlock_wait_cycles":2432,"thread_start_us":144,"threads_started":1}
I20260619 12:40:03.798460 23270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:40:03 dist-test-slave-kdrb krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 12:40:03.804842 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.794872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35996 (local address 127.0.98.60:46145)
0619 12:40:03.795210 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.795218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.795232 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:40:03.795265 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.795267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.795322 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.795888 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.795967 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.796740 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.797773 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.798087 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.798204 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.800495 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:03.800501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:03.800505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:03.800533 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:03.803064 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.803622 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.803626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.803627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.803716 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.804018 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.804021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.804023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.804229 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:03.804327 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:03.804582 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 12:40:03.804692 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
I20260619 12:40:03.804833 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.794797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46145 (local address 127.0.0.1:35996)
0619 12:40:03.795011 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.795025 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:03.795098 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:03.795459 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:03.795462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:03.795652 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:03.795778 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.795785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.796867 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.796871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.797640 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.797650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.798204 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.798208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.798289 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.798802 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:03.798817 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:03.800367 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:03.803205 (+ 2838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.803209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.803211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.803509 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.803823 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.803826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.803828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.803917 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.804316 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:03.804319 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:03.804416 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 12:40:03.804519 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 12:40:03.804656 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
Jun 19 12:40:03 dist-test-slave-kdrb krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:40:03.805289 23194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } attempt: 3
Jun 19 12:40:03 dist-test-slave-kdrb krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 12:40:03.809859 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.800016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45008 (local address 127.0.98.62:39303)
0619 12:40:03.800139 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.800143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.800157 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:40:03.800246 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.800250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.800297 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.800395 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.801214 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.801733 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.802708 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.802933 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.803044 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.805950 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:03.805954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:03.805957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:03.805982 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:03.807661 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.808134 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.808140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.808145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.808246 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.808541 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.808544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.808546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.808742 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:03.808938 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:03.809553 (+ 615us) server_negotiation.cc:299] Negotiation successful
0619 12:40:03.809670 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:40:03.809880 23119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5: Bootstrap starting.
Jun 19 12:40:03 dist-test-slave-kdrb krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 12:40:03.812021 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.799060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.0.98.61:35683)
0619 12:40:03.799171 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.799177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.800617 (+ 1440us) server_negotiation.cc:407] Connection header received
0619 12:40:03.800722 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.800725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.800775 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.800881 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.801341 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.802199 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.803351 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.803764 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.803838 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.807108 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:03.807112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:03.807114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:03.807143 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:03.808844 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.809324 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.809327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.809329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.809425 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.809725 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.809728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.809729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.809949 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:03.810027 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:03.811716 (+ 1689us) server_negotiation.cc:299] Negotiation successful
0619 12:40:03.811850 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34,"spinlock_wait_cycles":8960}
I20260619 12:40:03.812533 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.798972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35683 (local address 127.0.0.1:50568)
0619 12:40:03.800532 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.800550 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:03.800642 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:03.800923 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:03.800927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:03.801080 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:03.801210 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.801217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.802351 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.802354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.803226 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.803235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.804010 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.804013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.804116 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.805159 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:03.805173 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:03.806742 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:03.808978 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.808980 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.808982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.809211 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.809537 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.809540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.809541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.809625 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.810586 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:03.810589 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:03.810680 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 12:40:03.812165 (+ 1485us) client_negotiation.cc:240] Negotiation successful
0619 12:40:03.812299 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1490}
I20260619 12:40:03.813298 23270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:40:03.814126 23119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5: Neither blocks nor log segments found. Creating new log.
I20260619 12:40:03.815117 23119 log.cc:824] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5: Log is configured to *not* fsync() on all Append() calls
I20260619 12:40:03.816632 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.805681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46145 (local address 127.0.0.1:36004)
0619 12:40:03.806380 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.806395 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:03.806489 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:03.806770 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:03.806777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:03.806977 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:03.807101 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.807109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.808234 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.808238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.809029 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.809038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.809769 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.809773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.809852 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.810315 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:03.810332 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:03.811741 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:03.815026 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.815029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.815032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.815276 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.815665 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.815668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.815670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.815764 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.816169 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:03.816172 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:03.816260 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 12:40:03.816346 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 12:40:03.816469 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605}
I20260619 12:40:03.817502 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.805947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36004 (local address 127.0.98.60:46145)
0619 12:40:03.806090 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.806094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.806479 (+ 385us) server_negotiation.cc:407] Connection header received
0619 12:40:03.806571 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.806576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.806630 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.807224 (+ 594us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.807322 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.808111 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.809171 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.809643 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.810195 (+ 552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.811879 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:03.811882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:03.811885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:03.811910 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:03.814920 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.815458 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.815461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.815463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.815570 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.815870 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.815873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.815874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.816061 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:03.817128 (+ 1067us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:03.817197 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 12:40:03.817315 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:40:03.817608 23119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5: No bootstrap required, opened a new log
Jun 19 12:40:03 dist-test-slave-kdrb krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 12:40:03 dist-test-slave-kdrb krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 12:40:03.821168 23194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa: Bootstrap starting.
I20260619 12:40:03.822695 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.813818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46145 (local address 127.0.0.1:36008)
0619 12:40:03.813913 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 12:40:03.813928 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:40:03.814002 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:40:03.814471 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:40:03.814475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:40:03.814705 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:40:03.814810 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.814820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.815788 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.815806 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.816559 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:40:03.816568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.817222 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.817225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:40:03.817303 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.817732 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:40:03.817775 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:40:03.819170 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:40:03.821275 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.821278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.821280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.821500 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.821758 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:40:03.821761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:40:03.821763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:40:03.821848 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:40:03.822220 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:40:03.822222 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:40:03.822291 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 12:40:03.822382 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 12:40:03.822521 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 12:40:03.822549 23119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } }
I20260619 12:40:03.822741 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.811329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.0.98.61:35683)
0619 12:40:03.811457 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.811461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.811493 (+ 32us) server_negotiation.cc:407] Connection header received
0619 12:40:03.811529 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.811531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.811581 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.811664 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.812537 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.813180 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.814647 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.815004 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.815079 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.818304 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:03.818308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:03.818311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:03.818337 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:03.820688 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.821225 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.821229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.821231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.821315 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.821631 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.821636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.821638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.821898 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:03.822081 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:03.822479 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 12:40:03.822604 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 12:40:03.822783 23119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dcf11ab4c3546109e7e9c5d41dea7b5, State: Initialized, Role: FOLLOWER
I20260619 12:40:03.823150 23119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } }
I20260619 12:40:03.823238 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.814032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36008 (local address 127.0.98.60:46145)
0619 12:40:03.814163 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.814170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.814183 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:40:03.814248 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.814253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.814301 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.814377 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.814931 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.815689 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.816674 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.817043 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.817127 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.819293 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:03.819297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:03.819300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:03.819320 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:03.821127 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.821602 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.821604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.821606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.821677 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.821932 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.821934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.821936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.822112 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:03.822190 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:03.822998 (+ 808us) server_negotiation.cc:299] Negotiation successful
0619 12:40:03.823106 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:40:03.824613 23194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa: Neither blocks nor log segments found. Creating new log.
I20260619 12:40:03.825078 23119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:40:03.825416 23194 log.cc:824] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa: Log is configured to *not* fsync() on all Append() calls
I20260619 12:40:03.826632 23270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6: Bootstrap starting.
I20260619 12:40:03.827567 23194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa: No bootstrap required, opened a new log
I20260619 12:40:03.829383 23270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6: Neither blocks nor log segments found. Creating new log.
I20260619 12:40:03.830227 23270 log.cc:824] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6: Log is configured to *not* fsync() on all Append() calls
W20260619 12:40:03.830375 23289 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:40:03.830483 23289 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:40:03.831537 23194 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } }
I20260619 12:40:03.831787 23194 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f656dbd32545d9beb385545308d5fa, State: Initialized, Role: FOLLOWER
I20260619 12:40:03.832158 23194 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } }
I20260619 12:40:03.833330 23194 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:40:03.833432 23270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6: No bootstrap required, opened a new log
I20260619 12:40:03.830014 23051 master_runner.cc:186] Error getting master registration for 127.0.98.61:35683: OK, instance_id { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" instance_seqno: 1781872803540777 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:40:03.834578 23051 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:40:03.835002 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } } }
I20260619 12:40:03.835136 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:40:03.836949 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:40:03.835776 23292 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } } }
I20260619 12:40:03.838297 23292 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:40:03.838377 23093 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45008
I20260619 12:40:03.838610 23093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 12:40:03.838728 23093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [term 0 FOLLOWER]: Advancing to term 1
W20260619 12:40:03.839057 23303 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:40:03.839186 23303 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:40:03.840184 23093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } }
I20260619 12:40:03.840719 23270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } }
I20260619 12:40:03.840973 23270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aaeec22e90446bd9b70fc9b8b8682f6, State: Initialized, Role: FOLLOWER
I20260619 12:40:03.841197 23093 leader_election.cc:288] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [CANDIDATE]: Term 1 election: Requested vote from peers c0f656dbd32545d9beb385545308d5fa (127.0.98.61:35683), 5aaeec22e90446bd9b70fc9b8b8682f6 (127.0.98.60:46145)
I20260619 12:40:03.841353 23270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } }
I20260619 12:40:03.841645 23168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c0f656dbd32545d9beb385545308d5fa"
I20260619 12:40:03.841909 23168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:40:03.843302 23168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dcf11ab4c3546109e7e9c5d41dea7b5 in term 1.
I20260619 12:40:03.843420 23244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6"
I20260619 12:40:03.843683 23244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:40:03.843758 23069 leader_election.cc:302] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dcf11ab4c3546109e7e9c5d41dea7b5, c0f656dbd32545d9beb385545308d5fa; no voters:
I20260619 12:40:03.844105 23277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:40:03.844340 23277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [term 1 LEADER]: Becoming Leader. State: Replica: 6dcf11ab4c3546109e7e9c5d41dea7b5, State: Running, Role: LEADER
I20260619 12:40:03.844734 23277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } }
I20260619 12:40:03.845600 23244 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dcf11ab4c3546109e7e9c5d41dea7b5 in term 1.
I20260619 12:40:03.842720 23126 master_runner.cc:186] Error getting master registration for 127.0.98.60:46145: OK, instance_id { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" instance_seqno: 1781872803772475 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:40:03.846148 23126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:40:03.843367 23270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:40:03.846907 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dcf11ab4c3546109e7e9c5d41dea7b5. Latest consensus state: current_term: 1 leader_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } } }
I20260619 12:40:03.847048 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [sys.catalog]: This master's current role is: LEADER
I20260619 12:40:03.847458 23314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:40:03.851657 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } } }
I20260619 12:40:03.851809 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 12:40:03.852341 23323 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:40:03.852449 23323 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:40:03.852759 23314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:40:03.858222 23244 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6 [term 1 FOLLOWER]: Refusing update from remote peer 6dcf11ab4c3546109e7e9c5d41dea7b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:40:03.858449 23168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa [term 1 FOLLOWER]: Refusing update from remote peer 6dcf11ab4c3546109e7e9c5d41dea7b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:40:03.858917 23277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:40:03.859149 23306 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:40:03.860134 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.848254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45018 (local address 127.0.98.62:39303)
0619 12:40:03.848431 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.848438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.850392 (+ 1954us) server_negotiation.cc:407] Connection header received
0619 12:40:03.850590 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.850594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.850642 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.850692 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.851776 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.852573 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.853754 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.854086 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.854164 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.856354 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:03.856370 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:03.856373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:03.856399 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:03.858161 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.858698 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.858702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.858704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.858790 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.859116 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.859119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.859121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.859332 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:03.859462 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:03.859872 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 12:40:03.859992 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 12:40:03.862936 23292 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dcf11ab4c3546109e7e9c5d41dea7b5. Latest consensus state: current_term: 1 leader_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } } }
I20260619 12:40:03.863076 23292 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:40:03.865696 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } } }
I20260619 12:40:03.865833 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [sys.catalog]: This master's current role is: LEADER
I20260619 12:40:03.866144 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dcf11ab4c3546109e7e9c5d41dea7b5. Latest consensus state: current_term: 1 leader_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } } }
I20260619 12:40:03.866282 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:40:03.866740 23292 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } } }
I20260619 12:40:03.866884 23292 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:40:03.867092 23328 mvcc.cc:205] Tried to move back new op lower bound from 7298551004596736000 to 7298551004552318976. Current Snapshot: MvccSnapshot[applied={T|T < 7298551004596736000}]
I20260619 12:40:03.869191 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.849765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36016 (local address 127.0.98.60:46145)
0619 12:40:03.849867 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.849873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.860669 (+ 10796us) server_negotiation.cc:407] Connection header received
0619 12:40:03.860833 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.860836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.860885 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.860934 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.861802 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.862419 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.863435 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.863776 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.863854 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.865623 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:03.865627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:03.865630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:03.865657 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:03.867406 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.867903 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.867907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.867909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.867994 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.868280 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.868285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.868287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.868473 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:03.868613 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:03.868903 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 12:40:03.869022 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 12:40:03.869459 23330 mvcc.cc:205] Tried to move back new op lower bound from 7298551004596736000 to 7298551004552318976. Current Snapshot: MvccSnapshot[applied={T|T < 7298551004596736000}]
I20260619 12:40:03.871246 23314 catalog_manager.cc:1347] Generated new cluster ID: e94df5f8bd7f44099426745e1946d3d3
I20260619 12:40:03.871325 23314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:40:03.876958 23329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } } }
I20260619 12:40:03.877137 23329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5 [sys.catalog]: This master's current role is: LEADER
I20260619 12:40:03.878314 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcf11ab4c3546109e7e9c5d41dea7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39303 } } peers { permanent_uuid: "c0f656dbd32545d9beb385545308d5fa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35683 } } peers { permanent_uuid: "5aaeec22e90446bd9b70fc9b8b8682f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46145 } } }
I20260619 12:40:03.878453 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:40:03.887897 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:03.849062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.0.98.61:35683)
0619 12:40:03.849183 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.849187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.859581 (+ 10394us) server_negotiation.cc:407] Connection header received
0619 12:40:03.863340 (+ 3759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.863344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.863394 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.863446 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.865992 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.866678 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.875650 (+ 8972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:03.876005 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:03.876144 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:03.883754 (+ 7610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:40:03.883759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:40:03.883763 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:40:03.883817 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:40:03.885731 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.886321 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.886326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.886328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.886422 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:40:03.886739 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:40:03.886745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:40:03.886749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:40:03.886984 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:40:03.887106 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:03.887585 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 12:40:03.887717 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:40:03.888576 23314 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:40:03.889984 23314 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:40:03.911615 23314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dcf11ab4c3546109e7e9c5d41dea7b5: Generated new TSK 0
I20260619 12:40:03.912679 23314 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:40:03.939724 23123 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:03.936914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45024 (local address 127.0.98.62:39303)
0619 12:40:03.937041 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.937045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.938502 (+ 1457us) server_negotiation.cc:407] Connection header received
0619 12:40:03.938731 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.938735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.938790 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.938844 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.939572 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45024: BlockingRecv error: recv got EOF from 127.0.0.1:45024 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 12:40:03.941628 23275 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:03.936456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36026 (local address 127.0.98.60:46145)
0619 12:40:03.936587 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.936591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.940207 (+ 3616us) server_negotiation.cc:407] Connection header received
0619 12:40:03.940679 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.940682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.940734 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.940788 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.941503 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36026: BlockingRecv error: recv got EOF from 127.0.0.1:36026 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 12:40:03.942384 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:03.937678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.0.98.61:35683)
0619 12:40:03.937784 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.937788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.940407 (+ 2619us) server_negotiation.cc:407] Connection header received
0619 12:40:03.940916 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.940919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.940965 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.941014 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.942264 (+ 1250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50584: BlockingRecv error: recv got EOF from 127.0.0.1:50584 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 12:40:03.946972 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:03.945450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.0.98.61:35683)
0619 12:40:03.945556 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.945560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.946030 (+ 470us) server_negotiation.cc:407] Connection header received
0619 12:40:03.946251 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.946254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.946334 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.946384 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.946867 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50598: BlockingRecv error: recv got EOF from 127.0.0.1:50598 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 12:40:03.951937 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:03.950370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50614 (local address 127.0.98.61:35683)
0619 12:40:03.950469 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.950473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.950994 (+ 521us) server_negotiation.cc:407] Connection header received
0619 12:40:03.951168 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.951171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.951226 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.951277 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.951838 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50614: BlockingRecv error: recv got EOF from 127.0.0.1:50614 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 12:40:03.957355 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:03.955937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50618 (local address 127.0.98.61:35683)
0619 12:40:03.956035 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.956038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.956511 (+ 473us) server_negotiation.cc:407] Connection header received
0619 12:40:03.956699 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.956703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.956751 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.956799 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.957255 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50618: BlockingRecv error: recv got EOF from 127.0.0.1:50618 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 12:40:03.965256 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:03.963510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.0.98.61:35683)
0619 12:40:03.963693 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.963698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.964275 (+ 577us) server_negotiation.cc:407] Connection header received
0619 12:40:03.964478 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.964483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.964537 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.964593 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.965143 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50626: BlockingRecv error: recv got EOF from 127.0.0.1:50626 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260619 12:40:03.977342 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:03.975563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.0.98.61:35683)
0619 12:40:03.975685 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.975690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.976303 (+ 613us) server_negotiation.cc:407] Connection header received
0619 12:40:03.976518 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.976522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.976579 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.976639 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.977220 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50628: BlockingRecv error: recv got EOF from 127.0.0.1:50628 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 12:40:03.997942 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:03.995899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50638 (local address 127.0.98.61:35683)
0619 12:40:03.996036 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:03.996042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:03.996727 (+ 685us) server_negotiation.cc:407] Connection header received
0619 12:40:03.996996 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:03.997001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:03.997063 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:03.997135 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:03.997809 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50638: BlockingRecv error: recv got EOF from 127.0.0.1:50638 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 12:40:04.035449 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:04.033082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50654 (local address 127.0.98.61:35683)
0619 12:40:04.033268 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:04.033274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:04.034030 (+ 756us) server_negotiation.cc:407] Connection header received
0619 12:40:04.034441 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:04.034445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:04.034514 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:04.034586 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:04.035311 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50654: BlockingRecv error: recv got EOF from 127.0.0.1:50654 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260619 12:40:04.103641 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:04.102463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.0.98.61:35683)
0619 12:40:04.102622 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:04.102628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:04.102809 (+ 181us) server_negotiation.cc:407] Connection header received
0619 12:40:04.102909 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:04.102911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:04.102959 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:04.103036 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:04.103475 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50658: BlockingRecv error: recv got EOF from 127.0.0.1:50658 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 12:40:04.235208 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:04.234047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50662 (local address 127.0.98.61:35683)
0619 12:40:04.234166 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:04.234178 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:04.234340 (+ 162us) server_negotiation.cc:407] Connection header received
0619 12:40:04.234476 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:04.234479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:04.234541 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:04.234626 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:04.235041 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50662: BlockingRecv error: recv got EOF from 127.0.0.1:50662 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 12:40:04.494647 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:40:04.493538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.0.98.61:35683)
0619 12:40:04.493672 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:04.493675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:04.493748 (+ 73us) server_negotiation.cc:407] Connection header received
0619 12:40:04.493877 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:04.493879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:04.493928 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:04.494013 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:40:04.494510 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50674: BlockingRecv error: recv got EOF from 127.0.0.1:50674 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:40:04.840601 23303 catalog_manager.cc:1259] Loaded cluster ID: e94df5f8bd7f44099426745e1946d3d3
I20260619 12:40:04.840742 23303 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa: loading cluster ID for follower catalog manager: success
I20260619 12:40:04.842442 23303 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa: acquiring CA information for follower catalog manager: success
I20260619 12:40:04.843706 23303 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c0f656dbd32545d9beb385545308d5fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:40:04.853696 23323 catalog_manager.cc:1259] Loaded cluster ID: e94df5f8bd7f44099426745e1946d3d3
I20260619 12:40:04.853783 23323 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6: loading cluster ID for follower catalog manager: success
I20260619 12:40:04.855520 23323 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6: acquiring CA information for follower catalog manager: success
I20260619 12:40:04.856704 23323 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5aaeec22e90446bd9b70fc9b8b8682f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:40:05.013055 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:05.008876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50688 (local address 127.0.98.61:35683)
0619 12:40:05.009125 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:05.009130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:05.009252 (+ 122us) server_negotiation.cc:407] Connection header received
0619 12:40:05.009449 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:05.009453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:05.009582 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:05.009705 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:40:05.009991 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:05.010526 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:05.011432 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:05.011701 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:05.011822 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:05.012168 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:40:05.012634 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:40:05.012779 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:05.012821 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 12:40:05.012891 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
I20260619 12:40:05.020105 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:05.016162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.0.98.60:46145)
0619 12:40:05.016418 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:05.016424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:05.016440 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:40:05.016635 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:05.016638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:05.016757 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:05.016903 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:40:05.017125 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:05.017644 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:05.018507 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:05.018742 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:05.018855 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:05.019170 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:40:05.019691 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:40:05.019832 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:05.019874 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 12:40:05.019933 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":80,"threads_started":1}
I20260619 12:40:05.027824 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:05.023245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.0.98.62:39303)
0619 12:40:05.023647 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:05.023652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:05.023667 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:40:05.023887 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:05.023891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:05.024087 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:05.024211 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:40:05.024688 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:05.025229 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:05.026268 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:05.026629 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:05.026742 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:05.027067 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:40:05.027341 (+ 274us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:40:05.027448 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:05.027571 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 12:40:05.027632 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":72,"threads_started":1}
I20260619 12:40:05.028182 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:05.023340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.0.98.60:46145)
0619 12:40:05.023496 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:05.023501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:05.023754 (+ 253us) server_negotiation.cc:407] Connection header received
0619 12:40:05.023947 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:05.023950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:05.024013 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:05.024063 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:40:05.024454 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:05.025292 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:05.026783 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:05.027135 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:05.027201 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:05.027712 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:40:05.027787 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:40:05.027833 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:05.028005 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 12:40:05.028061 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:40:05.028451 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:40:05.023794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50692 (local address 127.0.98.61:35683)
0619 12:40:05.023943 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 12:40:05.023947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:40:05.024059 (+ 112us) server_negotiation.cc:407] Connection header received
0619 12:40:05.024162 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:40:05.024168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:40:05.024225 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:40:05.024296 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:40:05.024526 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:05.025296 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:05.026783 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:40:05.027121 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:40:05.027201 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:40:05.027712 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:40:05.027787 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:40:05.027833 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 12:40:05.028256 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 12:40:05.028320 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 12:40:05.031343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 23051
I20260619 12:40:05.038031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 23126
I20260619 12:40:05.044129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGObR4d/build/debug/bin/kudu with pid 23201
2026-06-19T12:40:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3098 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6187 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (288065 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST